Visualizing 13 Million DNA Barcode Records: A Global Map of Malaise Trap Biodiversity

The Arthropod Biodiversity Monitoring Dashboard is a map of 2,676 Malaise trap sites around the world, each one a window into a local flying insect community as recorded through DNA barcoding. Select any site and you get the species composition, the sampling history, how diversity accumulates with effort, and which species are found nowhere else in the global Malaise trap network. Navigate between sites and you can follow threads of compositional similarity across continents, or watch how community composition has shifted year over year.

All of it comes from a single public data release: the BOLD Systems database as it stood on 17 April 2026. This article explains what that data contains, the eleven processing steps that convert it into the dashboard database, and the technical architecture that delivers it to the browser.

The source: BOLD Public 17-Apr-2026

BOLD Systems — the Barcode of Life Data System, maintained by the Centre for Biodiversity Genomics at the University of Guelph — is the world's largest repository of DNA barcodes. Every record in the system represents a specimen: a physical organism that was collected, identified, sequenced at a standard barcode locus (typically CO1 for animals), and uploaded with associated metadata.

BOLD releases periodic public snapshots of its entire holdings. The 17 April 2026 snapshot is distributed as a tab-separated file with 76 fields per record, covering taxonomy (phylum through subspecies), geography (country, province, coordinates, elevation, ecoregion), collection context (institution, field identifier, sampling protocol, collection dates, habitat), and sequence information (nucleotide sequence, primer pair, trace information, BIN assignment). The raw file spans tens of millions of records across animal, plant, and fungal kingdoms.

The dashboard uses a small, tightly filtered slice of this — only records collected with Malaise traps, with valid coordinates and BIN assignments, and with complete collection date records. Before we reach that filtering, the data goes through several preparation steps.

Step 1 — Field reduction

The first pass strips 48 fields that are not needed for any downstream computation: raw nucleotide sequences, primer information, funding sources, sequence upload timestamps, and similar fields. Each record goes from 76 fields to 28, producing a leaner working file while keeping everything needed for geographic, taxonomic, and temporal analysis.

No records are dropped at this step — only columns.

Step 2 — Deduplication and aggregation

Many records in BOLD represent repeat observations of the same organism at the same place and time under the same collection event. The extracted file is grouped on 17 key fields — including rounded coordinates, BIN URI, full taxonomy down to subspecies, sampling protocol, collection dates, field identifier, museum ID, and institution. Duplicate rows within each group are collapsed into a single row with a count column recording how many original records it represents.

This step achieves a 10–50× compression of the row count, dramatically reducing computational load for all subsequent steps while preserving the complete biological signal. No biological information is lost.

Step 3 — Malaise trap filtering

This is the most consequential filtering step. From the entire BOLD record set, only records that pass all five of the following criteria are carried forward:

1. Sampling protocol must contain the word "malaise" (case-insensitive), isolating records collected specifically using passive tent-shaped intercept traps designed for flying insects.
2. Valid coordinates — latitude between −90 and 90, longitude between −180 and 180. Records with missing, null, or out-of-range coordinates are excluded.
3. Valid BIN URI — the record must have a non-empty Barcode Index Number assignment. BINs are algorithmically assigned species-level proxies; records without one cannot contribute to diversity estimates.
4. Complete collection dates — both a start date and end date must be present in valid ISO format.
5. Date range — collection dates must fall between 1990 and 2026, and the end date must not precede the start date.

Records failing any one of these five criteria are dropped entirely. The vast majority of BOLD records come from sweep nets, light traps, Berlese funnels, hand collection, and pitfall traps — none of which qualify. Within Malaise records, a further fraction are lost due to missing coordinates or missing BIN assignments, which are common in older or less thoroughly databased collection events.

Step 4 — Site construction

Coordinates that pass the filter are rounded to three decimal places (approximately 100 metres of precision) and used to construct a site identifier in the format {lat:.3f}_{lon:.3f} (for example, -35.261_149.059). All records falling at the same rounded coordinate are treated as belonging to the same site.

For each site, a comprehensive set of statistics is computed covering temporal coverage (first and last collection date, span in days and years, number of active years, maximum gap between active years, peak activity month), sampling effort (distinct field identifiers as a proxy for trap deployments, estimated maximum simultaneous traps, number of discrete collection events, median deployment duration), biological richness (total specimen count, total BIN count, orders and families represented, specimens and BINs per year), and metadata (modal country, province, site name, ecoregion, biome, realm, and elevation derived from the most frequent values across all records at that site).

Steps 5–10 — Analytical computation

With sites constructed, six further analytical passes produce the data that drives each panel in the viewer.

Compositional analysis (Step 5). Records are grouped by taxonomic order and family to produce an order-level composition profile for each site. Families are ranked by BIN count within each order and capped at 12 per order, with any remainder aggregated into an "Other" group. This is what powers the sunburst chart.

Phenology timeseries (Step 6). Collection records are aggregated by calendar month (YYYY-MM) to produce a monthly activity timeseries: specimens collected and distinct BINs detected per month. This drives the sampling activity chart.

Rarefaction curves (Step 7). Rarefaction quantifies how BIN richness accumulates as sampling effort increases, enabling fair comparison between sites with very different numbers of total specimens. The method used is Hurlbert (1971) exact hypergeometric rarefaction, computed in log-space to avoid numerical overflow:

E(S_n) = S − Σ_i [ C(N − n_i, n) / C(N, n) ]

Site similarity (Step 8). Pairwise Jaccard similarity is computed for each site against all other sites using their full BIN inventories. The top 10 most compositionally similar sites are stored per site — these are the sites navigated by the arrow keys in the dashboard. Geographic neighbours (sites within 200 km with at least two weeks of temporal overlap) are also computed as an alternative navigation mode.

Temporal turnover (Step 9). For sites with three or more years of data, year-to-year compositional turnover is quantified using mean pairwise Sørensen dissimilarity. The result is a single value between 0 (completely stable community year over year) and 1 (completely different community each year), shown in each site panel.

Exclusive BINs (Step 10). A BIN is classified as "site-exclusive" if it was detected at exactly one site across the entire filtered dataset — a global singleton in the context of the Malaise trap network. The count of exclusive BINs is tallied by taxonomic order for each site. For sites with at least one exclusive BIN, specimen images are retrieved from the CAOS image server via batch API queries, with up to three images per BIN.

Step 11 — Search index and final database

A compact SQLite index maps every unique BIN URI and species name in the dataset to the sites where it was detected. This powers the BIN / species search feature in the dashboard controls, allowing users to look up any barcode or name and immediately see where it appears in the global Malaise trap network.

All outputs are assembled into two SQLite databases. The main database (malaise-viewer.sqlite) holds the bootstrap payload — the full GeoJSON feature collection of all sites, the complete list of active months, and a year-to-site index for the timeline filter — alongside per-site payloads for composition, timeseries, rarefaction, and exclusive BIN data. The companion database (malaise-search.sqlite) holds the taxa search index separately to keep startup fast.

Data volumes

Specimen counts per site range from the filtering minimum up to roughly 30,000. BIN richness per site ranges from a handful to over 2,000. The smallest sites have a few hundred specimens collected over a single season; the largest span multiple years and represent continuous long-term monitoring programs.

Dashboard architecture

The viewer is a static HTML page. It loads MapLibre GL JS from a CDN and initializes a base map, then immediately fetches the bootstrap payload from the API — a single JSON object containing all 2,676 site locations, their summary statistics, and the active month index for the timeline filter. That GeoJSON is rendered as a clustered point layer; at higher zoom levels it resolves into individual sites.

No per-site chart data is loaded until a user selects a site. Clicking a map marker (or searching for a BIN / species name and following the result) triggers an API call to /api/malaise/sites/{siteId}. The response contains everything the right-hand panel needs: composition, timeseries, rarefaction curve, exclusive BIN counts, and the top 10 compositional neighbours. A second, deferred call fetches exclusive BIN specimen images if the user opens the images panel.

This lazy-loading pattern keeps the initial page load under one second on a typical connection, regardless of how large the full dataset grows. The heavy computation — Jaccard similarity across all site pairs, rarefaction at 15 sample sizes, Sørensen turnover across all year pairs — all happened at build time and is stored as precomputed payloads in SQLite. The server does no computation at query time; it reads a single row from a database and returns it.

The build pipeline separates processing from serving

The data preparation pipeline runs on a build machine with a Python environment, the full BOLD snapshot on disk, and the compute time to process tens of millions of records. This is a different machine from the production server. The heavy work — steps 1 through 11 — runs once on the build machine and produces two finished SQLite files. Only those SQLite files are shipped to the production server.

The production environment is deliberately lean: a lightweight Fastify API process, the two SQLite files, and static HTML and assets served by Nginx. No Python, no pandas, no SciPy, no data pipeline tooling on the server at all. The 1 GB Ubuntu instance running the site has no awareness of the BOLD snapshot it originated from.

The viewer shell itself is also built, not served raw. A build script transforms the source HTML with a series of targeted string replacements before output lands in the static export directory. The most significant transformation replaces the local selectSite() function — which reads from local data files during development — with an async version that calls the API. Chart rendering logic, search index wiring, and image payload fetching are all injected at build time. This means the deployed viewer has no awareness of how it was assembled and no dependency on build tooling at runtime.

What the dashboard does not show

The dashboard is not a complete picture of arthropod diversity. Several important caveats are worth stating plainly.

Non-Malaise trap records are excluded. The vast majority of arthropod barcodes in BOLD — collected by sweep net, light trap, Berlese funnel, hand collection, and other methods — are not included. Malaise traps have a strong sampling bias toward flying Diptera and Hymenoptera. Ground-dwelling beetles, soil-dwelling Collembola, nocturnal Lepidoptera, and many other groups are systematically underrepresented or absent, depending on the site.

Records without BIN assignments are excluded. Sequences that have not been algorithmically grouped into a BIN are dropped. This disproportionately affects older records and those from taxonomically underrepresented lineages where barcode divergence thresholds are uncertain. A site with many unassigned sequences will appear to have lower diversity than it actually has.

Records without complete collection dates are excluded. This affects effort estimation across the board. Sites where trap deployment was not recorded with both a start and end date cannot contribute to phenology or temporal turnover analysis.

BINs are not species. The dashboard works with Barcode Index Numbers, not formally described species. BINs are algorithmically constructed groupings that approximate species-level boundaries under standard divergence thresholds. They correlate well with described species but do not map one-to-one. A single BIN may contain multiple cryptic species; conversely, conspecific populations with high intraspecific variation may be split across multiple BINs.

Specimen counts are not trap catch abundance. BOLD records reflect the number of individual specimens barcoded, not the total insects caught in the trap. Subsampling protocols vary widely across programs and institutions. A site that barcodes 50 individuals per trap deployment is not directly comparable to one that barcodes 5,000 — the latter is almost certainly not catching 100× more insects, it is simply processing more specimens per trap.

These limitations are intrinsic to the underlying data rather than choices we made in building the dashboard. The goal is to make the existing Malaise trap barcode record navigable and analytically useful — not to represent it as more complete than it is.

---