⚡ go-batcher

High-performance batch processing for Go applications

CI / CD	Quality & Security	Docs & Meta	Community

🗂️ Table of Contents

• What's Inside
• Installation
• Documentation
• Examples & Tests
• Benchmarks
• Code Standards
• AI Compliance
• Maintainers
• Contributing
• License

🎯 What's Inside

Lightning-Fast Batch Processing in Action

package main

import (
    "fmt"
    "time"
    "github.com/bsv-blockchain/go-batcher"
)

func main() {
    // Create a batcher that processes items every 100ms or when batch size hits 1000
    b := batcher.New[string](
        1000,                                        // batch size
        100*time.Millisecond,                        // timeout interval
        func(batch []*string) {                      // processor function
            fmt.Printf("⚡ Processing %d items in one go!\n", len(batch))
            // Your batch processing logic here
            for _, item := range batch {
                fmt.Printf("Processing: %s\n", *item)
            }
        },
        true, // background processing
    )

    // Feed items - they'll be intelligently batched
    for i := 0; i < 5000; i++ {
        item := fmt.Sprintf("item-%d", i)
        b.Put(&item)
    }

    // Process any remaining items before shutdown
    b.Trigger()
    // Note: The batcher worker runs indefinitely - use context cancellation for cleanup
}

Constructor Variants

The go-batcher library provides several constructor options to fit different use cases:

// Basic batcher - simple batching with size and timeout triggers
b := batcher.New[string](100, time.Second, processFn, true)

// With slice pooling - reduces memory allocations for high-throughput scenarios
b := batcher.NewWithPool[string](100, time.Second, processFn, true)

// With automatic deduplication - filters duplicate items within a 1-minute window
b := batcher.NewWithDeduplication[string](100, time.Second, processFn, true)

// Combined pooling and deduplication - maximum performance with duplicate filtering
b := batcher.NewWithDeduplicationAndPool[string](100, time.Second, processFn, true)

Why You'll Love This Batcher

• Blazing Performance – Process millions of items with minimal overhead (benchmarks: 135 ns/op)
• Smart Batching – Auto-groups by size or time interval, whichever comes first
• Optional Deduplication – Built-in dedup variant ensures each item is processed only once within a time window
• Memory Pool Optimization – Optional slice pooling reduces GC pressure in high-throughput scenarios
• Thread-Safe by Design – Concurrent Put() from multiple goroutines without worry
• Time-Partitioned Storage – Efficient memory usage with automatic cleanup (dedup variant)
• Minimal Dependencies – Pure Go with only essential external dependencies
• Flexible Configuration – Multiple constructor variants for different use cases
• Production-Ready – Battle-tested with full test coverage and benchmarks

Perfect for high-throughput scenarios like log aggregation, metrics collection, event processing, or any situation where you need to efficiently batch operations for downstream systems.

📦 Installation

go-batcher requires a supported release of Go.

go get -u github.com/bsv-blockchain/go-batcher

📚 Documentation

• API Reference – Dive into the godocs at pkg.go.dev/github.com/bsv-blockchain/go-batcher
• Usage Examples – Browse practical patterns either the examples directory or view the example functions
• Benchmarks – Check the latest numbers in the benchmark results
• Test Suite – Review both the unit tests and fuzz tests (powered by testify)

Development Build Commands

Get the MAGE-X build tool for development:

go install github.com/mrz1836/mage-x/cmd/magex@latest

View all build commands

magex help

Repository Features

• Continuous Integration on Autopilot with GitHub Actions – every push is built, tested, and reported in minutes.
• Pull‑Request Flow That Merges Itself thanks to auto‑merge and hands‑free Dependabot auto‑merge.
• One‑Command Builds powered by battle‑tested MAGE-X targets for linting, testing, releases, and more.
• First‑Class Dependency Management using native Go Modules.
• Uniform Code Style via gofumpt plus zero‑noise linting with golangci‑lint.
• Confidence‑Boosting Tests with testify, the Go race detector, crystal‑clear HTML coverage snapshots, and automatic uploads to Codecov.
• Hands‑Free Releases delivered by GoReleaser whenever you create a new Tag.
• Relentless Dependency & Vulnerability Scans via Dependabot, Nancy and govulncheck.
• Security Posture by Default with CodeQL, OpenSSF Scorecard and secret‑leak detection via gitleaks.
• Automatic Syndication to pkg.go.dev on every release for instant godoc visibility.
• Polished Community Experience using rich templates for Issues & PRs.
• All the Right Meta Files (LICENSE, CONTRIBUTING.md, CODE_OF_CONDUCT.md, SUPPORT.md, SECURITY.md) pre‑filled and ready.
• Code Ownership clarified through a CODEOWNERS file, keeping reviews fast and focused.
• Zero‑Noise Dev Environments with tuned editor settings (.editorconfig) plus curated ignore files for VS Code, Docker, and Git.
• Label Sync Magic: your repo labels stay in lock‑step with .github/labels.yml.
• Friendly First PR Workflow – newcomers get a warm welcome thanks to a dedicated workflow.
• Standards‑Compliant Docs adhering to the standard‑readme spec.
• Instant Cloud Workspaces via Gitpod – spin up a fully configured dev environment with automatic linting and tests.
• Out‑of‑the‑Box VS Code Happiness with a preconfigured Go workspace and .vscode folder with all the right settings.
• Optional Release Broadcasts to your community via Slack, Discord, or Twitter – plug in your webhook.
• AI Compliance Playbook – machine‑readable guidelines (AGENTS.md, CLAUDE.md, .cursorrules, sweep.yaml) keep ChatGPT, Claude, Cursor & Sweep aligned with your repo's rules.
• Go-Pre-commit System - High-performance Go-native pre-commit hooks with 17x faster execution—run the same formatting, linting, and tests before every commit, just like CI.
• Zero Python Dependencies - Pure Go implementation with environment-based configuration via .env.base.
• DevContainers for Instant Onboarding – Launch a ready-to-code environment in seconds with VS Code DevContainers and the included .devcontainer.json config.

Library Deployment

This project uses goreleaser for streamlined binary and library deployment to GitHub. To get started, install it via:

brew install goreleaser

The release process is defined in the .goreleaser.yml configuration file.

Then create and push a new Git tag using:

magex version:bump push=true bump=patch branch=master

This process ensures consistent, repeatable releases with properly versioned artifacts and citation metadata.

Pre-commit Hooks

Set up the Go-Pre-commit System to run the same formatting, linting, and tests defined in AGENTS.md before every commit:

go install github.com/mrz1836/go-pre-commit/cmd/go-pre-commit@latest
go-pre-commit install

The system is configured via .env.base and can be customized using also using .env.custom and provides 17x faster execution than traditional Python-based pre-commit hooks. See the complete documentation for details.

GitHub Workflows

🎛️ The Workflow Control Center

All GitHub Actions workflows in this repository are powered by a single configuration files – your one-stop shop for tweaking CI/CD behavior without touching a single YAML file! 🎯

Configuration Files:

• .env.base – Default configuration that works for most Go projects
• .env.custom – Optional project-specific overrides

This magical file controls everything from:

• ⚙️ Go version matrix (test on multiple versions or just one)
• 🏃 Runner selection (Ubuntu or macOS, your wallet decides)
• 🔬 Feature toggles (coverage, fuzzing, linting, race detection, benchmarks)
• 🛡️ Security tool versions (gitleaks, nancy, govulncheck)
• 🤖 Auto-merge behaviors (how aggressive should the bots be?)
• 🏷️ PR management rules (size labels, auto-assignment, welcome messages)

Workflow Name	Description
auto-merge-on-approval.yml	Automatically merges PRs after approval and all required checks, following strict rules.
codeql-analysis.yml	Analyzes code for security vulnerabilities using GitHub CodeQL.
dependabot-auto-merge.yml	Automatically merges Dependabot PRs that meet all requirements.
fortress.yml	Runs the GoFortress security and testing workflow, including linting, testing, releasing, and vulnerability checks.
pull-request-management.yml	Labels PRs by branch prefix, assigns a default user if none is assigned, and welcomes new contributors with a comment.
scorecard.yml	Runs OpenSSF Scorecard to assess supply chain security.
stale.yml	Warns about (and optionally closes) inactive issues and PRs on a schedule or manual trigger.
sync-labels.yml	Keeps GitHub labels in sync with the declarative manifest at .github/labels.yml.

Updating Dependencies

To update all dependencies (Go modules, linters, and related tools), run:

magex deps:update

This command ensures all dependencies are brought up to date in a single step, including Go modules and any tools managed by MAGE-X. It is the recommended way to keep your development environment and CI in sync with the latest versions.

🧪 Examples & Tests

All unit tests and examples run via GitHub Actions and use Go version 1.24.x. View the configuration file.

Run all tests (fast):

magex test

Run all tests with race detector (slower):

magex test:race

⚡ Benchmarks

Run the Go benchmarks:

magex bench

Benchmark Results

Benchmark	Description	ns/op	B/op	allocs/op
BenchmarkBatcherPut	Basic Put operation	135.1	8	0
BenchmarkBatcherPutParallel	Concurrent Put operations	310.0	9	0
BenchmarkPutComparison/Put	Put operation (non-blocking)	300.7	9	0
BenchmarkPutComparison/PutWithPool	Put with slice pooling	309.9	1	0
BenchmarkWithPoolComparison/Batcher	Standard batcher	171.2	18	1
BenchmarkWithPoolComparison/WithPool	Pooled batcher	184.0	9	1
BenchmarkTimePartitionedMapSet	Map Set operation (bloom filter)	366.7	147	6
BenchmarkTimePartitionedMapGet	Map Get operation (bloom filter)	80.5	39	2
BenchmarkBatcherWithDedupPut	Put with deduplication	740.1	166	7
BenchmarkBatcher	Full batch processing (1M items)	1,081ms	710MB	1.9M
BenchmarkBatcherWithDeduplication	Deduplication processing	90.7	13	0

Performance benchmarks for the core functions in this library, executed on an Apple M1 Max (ARM64). The benchmarks demonstrate excellent performance with minimal allocations for basic operations.

🛠️ Code Standards

Read more about this Go project's code standards.

🤖 AI Compliance

This project documents expectations for AI assistants using a few dedicated files:

• AGENTS.md — canonical rules for coding style, workflows, and pull requests used by Codex.
• CLAUDE.md — quick checklist for the Claude agent.
• .cursorrules — machine-readable subset of the policies for Cursor and similar tools.
• sweep.yaml — rules for Sweep, a tool for code review and pull request management.

Edit AGENTS.md first when adjusting these policies, and keep the other files in sync within the same pull request.

👥 Maintainers

MrZ	Siggi

🤝 Contributing

View the contributing guidelines and please follow the code of conduct.

How can I help?

All kinds of contributions are welcome 🙌! The most basic way to show your support is to star 🌟 the project, or to raise issues 💬.

📝 License