High-performance batch processing for Go applications
CI / CD | Quality & Security | Docs & Meta | Community |
---|---|---|---|
|
|
|
|
- What's Inside
- Installation
- Documentation
- Examples & Tests
- Benchmarks
- Code Standards
- AI Compliance
- Maintainers
- Contributing
- License
package main
import (
"fmt"
"time"
"github.com/bsv-blockchain/go-batcher"
)
func main() {
// Create a batcher that processes items every 100ms or when batch size hits 1000
b := batcher.New[string](
1000, // batch size
100*time.Millisecond, // timeout interval
func(batch []*string) { // processor function
fmt.Printf("⚡ Processing %d items in one go!\n", len(batch))
// Your batch processing logic here
for _, item := range batch {
fmt.Printf("Processing: %s\n", *item)
}
},
true, // background processing
)
// Feed items - they'll be intelligently batched
for i := 0; i < 5000; i++ {
item := fmt.Sprintf("item-%d", i)
b.Put(&item)
}
// Process any remaining items before shutdown
b.Trigger()
// Note: The batcher worker runs indefinitely - use context cancellation for cleanup
}
The go-batcher
library provides several constructor options to fit different use cases:
// Basic batcher - simple batching with size and timeout triggers
b := batcher.New[string](100, time.Second, processFn, true)
// With slice pooling - reduces memory allocations for high-throughput scenarios
b := batcher.NewWithPool[string](100, time.Second, processFn, true)
// With automatic deduplication - filters duplicate items within a 1-minute window
b := batcher.NewWithDeduplication[string](100, time.Second, processFn, true)
// Combined pooling and deduplication - maximum performance with duplicate filtering
b := batcher.NewWithDeduplicationAndPool[string](100, time.Second, processFn, true)
- Blazing Performance – Process millions of items with minimal overhead (benchmarks: 135 ns/op)
- Smart Batching – Auto-groups by size or time interval, whichever comes first
- Optional Deduplication – Built-in dedup variant ensures each item is processed only once within a time window
- Memory Pool Optimization – Optional slice pooling reduces GC pressure in high-throughput scenarios
- Thread-Safe by Design – Concurrent Put() from multiple goroutines without worry
- Time-Partitioned Storage – Efficient memory usage with automatic cleanup (dedup variant)
- Minimal Dependencies – Pure Go with only essential external dependencies
- Flexible Configuration – Multiple constructor variants for different use cases
- Production-Ready – Battle-tested with full test coverage and benchmarks
Perfect for high-throughput scenarios like log aggregation, metrics collection, event processing, or any situation where you need to efficiently batch operations for downstream systems.
go-batcher requires a supported release of Go.
go get -u github.com/bsv-blockchain/go-batcher
- API Reference – Dive into the godocs at pkg.go.dev/github.com/bsv-blockchain/go-batcher
- Usage Examples – Browse practical patterns either the examples directory or view the example functions
- Benchmarks – Check the latest numbers in the benchmark results
- Test Suite – Review both the unit tests and fuzz tests (powered by
testify
)
Development Build Commands
Get the MAGE-X build tool for development:
go install github.com/mrz1836/mage-x/cmd/magex@latest
View all build commands
magex help
Repository Features
- Continuous Integration on Autopilot with GitHub Actions – every push is built, tested, and reported in minutes.
- Pull‑Request Flow That Merges Itself thanks to auto‑merge and hands‑free Dependabot auto‑merge.
- One‑Command Builds powered by battle‑tested MAGE-X targets for linting, testing, releases, and more.
- First‑Class Dependency Management using native Go Modules.
- Uniform Code Style via gofumpt plus zero‑noise linting with golangci‑lint.
- Confidence‑Boosting Tests with testify, the Go race detector, crystal‑clear HTML coverage snapshots, and automatic uploads to Codecov.
- Hands‑Free Releases delivered by GoReleaser whenever you create a new Tag.
- Relentless Dependency & Vulnerability Scans via Dependabot, Nancy and govulncheck.
- Security Posture by Default with CodeQL, OpenSSF Scorecard and secret‑leak detection via gitleaks.
- Automatic Syndication to pkg.go.dev on every release for instant godoc visibility.
- Polished Community Experience using rich templates for Issues & PRs.
- All the Right Meta Files (
LICENSE
,CONTRIBUTING.md
,CODE_OF_CONDUCT.md
,SUPPORT.md
,SECURITY.md
) pre‑filled and ready. - Code Ownership clarified through a CODEOWNERS file, keeping reviews fast and focused.
- Zero‑Noise Dev Environments with tuned editor settings (
.editorconfig
) plus curated ignore files for VS Code, Docker, and Git. - Label Sync Magic: your repo labels stay in lock‑step with .github/labels.yml.
- Friendly First PR Workflow – newcomers get a warm welcome thanks to a dedicated workflow.
- Standards‑Compliant Docs adhering to the standard‑readme spec.
- Instant Cloud Workspaces via Gitpod – spin up a fully configured dev environment with automatic linting and tests.
- Out‑of‑the‑Box VS Code Happiness with a preconfigured Go workspace and
.vscode
folder with all the right settings. - Optional Release Broadcasts to your community via Slack, Discord, or Twitter – plug in your webhook.
- AI Compliance Playbook – machine‑readable guidelines (AGENTS.md, CLAUDE.md, .cursorrules, sweep.yaml) keep ChatGPT, Claude, Cursor & Sweep aligned with your repo's rules.
- Go-Pre-commit System - High-performance Go-native pre-commit hooks with 17x faster execution—run the same formatting, linting, and tests before every commit, just like CI.
- Zero Python Dependencies - Pure Go implementation with environment-based configuration via .env.base.
- DevContainers for Instant Onboarding – Launch a ready-to-code environment in seconds with VS Code DevContainers and the included .devcontainer.json config.
Library Deployment
This project uses goreleaser for streamlined binary and library deployment to GitHub. To get started, install it via:
brew install goreleaser
The release process is defined in the .goreleaser.yml configuration file.
Then create and push a new Git tag using:
magex version:bump push=true bump=patch branch=master
This process ensures consistent, repeatable releases with properly versioned artifacts and citation metadata.
Pre-commit Hooks
Set up the Go-Pre-commit System to run the same formatting, linting, and tests defined in AGENTS.md before every commit:
go install github.com/mrz1836/go-pre-commit/cmd/go-pre-commit@latest
go-pre-commit install
The system is configured via .env.base and can be customized using also using .env.custom and provides 17x faster execution than traditional Python-based pre-commit hooks. See the complete documentation for details.
GitHub Workflows
All GitHub Actions workflows in this repository are powered by a single configuration files – your one-stop shop for tweaking CI/CD behavior without touching a single YAML file! 🎯
Configuration Files:
- .env.base – Default configuration that works for most Go projects
- .env.custom – Optional project-specific overrides
This magical file controls everything from:
- ⚙️ Go version matrix (test on multiple versions or just one)
- 🏃 Runner selection (Ubuntu or macOS, your wallet decides)
- 🔬 Feature toggles (coverage, fuzzing, linting, race detection, benchmarks)
- 🛡️ Security tool versions (gitleaks, nancy, govulncheck)
- 🤖 Auto-merge behaviors (how aggressive should the bots be?)
- 🏷️ PR management rules (size labels, auto-assignment, welcome messages)
Workflow Name | Description |
---|---|
auto-merge-on-approval.yml | Automatically merges PRs after approval and all required checks, following strict rules. |
codeql-analysis.yml | Analyzes code for security vulnerabilities using GitHub CodeQL. |
dependabot-auto-merge.yml | Automatically merges Dependabot PRs that meet all requirements. |
fortress.yml | Runs the GoFortress security and testing workflow, including linting, testing, releasing, and vulnerability checks. |
pull-request-management.yml | Labels PRs by branch prefix, assigns a default user if none is assigned, and welcomes new contributors with a comment. |
scorecard.yml | Runs OpenSSF Scorecard to assess supply chain security. |
stale.yml | Warns about (and optionally closes) inactive issues and PRs on a schedule or manual trigger. |
sync-labels.yml | Keeps GitHub labels in sync with the declarative manifest at .github/labels.yml . |
Updating Dependencies
To update all dependencies (Go modules, linters, and related tools), run:
magex deps:update
This command ensures all dependencies are brought up to date in a single step, including Go modules and any tools managed by MAGE-X. It is the recommended way to keep your development environment and CI in sync with the latest versions.
All unit tests and examples run via GitHub Actions and use Go version 1.24.x. View the configuration file.
Run all tests (fast):
magex test
Run all tests with race detector (slower):
magex test:race
Run the Go benchmarks:
magex bench
Benchmark | Description | ns/op | B/op | allocs/op |
---|---|---|---|---|
BenchmarkBatcherPut | Basic Put operation | 135.1 | 8 | 0 |
BenchmarkBatcherPutParallel | Concurrent Put operations | 310.0 | 9 | 0 |
BenchmarkPutComparison/Put | Put operation (non-blocking) | 300.7 | 9 | 0 |
BenchmarkPutComparison/PutWithPool | Put with slice pooling | 309.9 | 1 | 0 |
BenchmarkWithPoolComparison/Batcher | Standard batcher | 171.2 | 18 | 1 |
BenchmarkWithPoolComparison/WithPool | Pooled batcher | 184.0 | 9 | 1 |
BenchmarkTimePartitionedMapSet | Map Set operation (bloom filter) | 366.7 | 147 | 6 |
BenchmarkTimePartitionedMapGet | Map Get operation (bloom filter) | 80.5 | 39 | 2 |
BenchmarkBatcherWithDedupPut | Put with deduplication | 740.1 | 166 | 7 |
BenchmarkBatcher | Full batch processing (1M items) | 1,081ms | 710MB | 1.9M |
BenchmarkBatcherWithDeduplication | Deduplication processing | 90.7 | 13 | 0 |
Performance benchmarks for the core functions in this library, executed on an Apple M1 Max (ARM64). The benchmarks demonstrate excellent performance with minimal allocations for basic operations.
Read more about this Go project's code standards.
This project documents expectations for AI assistants using a few dedicated files:
- AGENTS.md — canonical rules for coding style, workflows, and pull requests used by Codex.
- CLAUDE.md — quick checklist for the Claude agent.
- .cursorrules — machine-readable subset of the policies for Cursor and similar tools.
- sweep.yaml — rules for Sweep, a tool for code review and pull request management.
Edit AGENTS.md
first when adjusting these policies, and keep the other files in sync within the same pull request.
![]() |
![]() |
---|---|
MrZ | Siggi |
View the contributing guidelines and please follow the code of conduct.
All kinds of contributions are welcome 🙌! The most basic way to show your support is to star 🌟 the project, or to raise issues 💬.