Most codebases do not fail because the developers lack talent. They fail because the team never established the structural patterns that prevent complexity from compounding. A service starts with clean function signatures and clear intent. Six months later, error handling is inconsistent, concurrency is ad hoc, configuration is scattered across environment variables and hardcoded defaults, and every new feature requires modifying five files that should not know about each other.
Patterns are not academic abstractions. They are the engineering vocabulary that makes systems predictable. When every service in a codebase uses the same error wrapping convention, the same graceful shutdown sequence, and the same dependency injection approach, a new team member can navigate any service after reading one. When patterns are absent, every service is a snowflake that requires its own orientation.
What follows is a catalog of one hundred patterns that I reach for repeatedly in Go and Python production systems. They span interface design, concurrency, resilience, data access, observability, testing, and architecture. Each pattern includes a focused code example — Go where the pattern is native to the language, Python where the idiom differs meaningfully, and both where the contrast is instructive.
This is not a tutorial. It is a reference. Read it end to end to build a mental model, or jump to the section that addresses the problem you are solving today.
Part I: Interface Design and Composition Link to heading
1. Small, Consumer-Owned Interfaces Link to heading
Define interfaces where they are consumed, not where they are implemented. A service that needs to read users should declare a two-method interface, not import a twenty-method interface from the repository package. Small interfaces are easier to mock, easier to satisfy, and resistant to shotgun changes.
// In the consumer package, not the provider package.
// The service declares exactly what it needs.
package userservice
type UserReader interface {
GetByID(ctx context.Context, id string) (User, error)
GetByEmail(ctx context.Context, email string) (User, error)
}
type Service struct {
users UserReader // not *postgres.UserRepository
}
# Python equivalent using Protocol — defined where consumed.
from typing import Protocol
class UserReader(Protocol):
def get_by_id(self, user_id: str) -> User | None: ...
def get_by_email(self, email: str) -> User | None: ...
class UserService:
def __init__(self, users: UserReader) -> None:
self._users = users
2. Context Propagation Everywhere Link to heading
Every function that performs I/O, crosses a service boundary, or might be cancelled should accept context.Context as its first parameter. Context carries deadlines, cancellation signals, and request-scoped values like trace IDs. Omitting it makes graceful shutdown impossible and distributed tracing incomplete.
// Context flows from HTTP handler through service to repository.
func (s *Service) ProcessOrder(ctx context.Context, orderID string) error {
order, err := s.orders.GetByID(ctx, orderID)
if err != nil {
return fmt.Errorf("fetch order %s: %w", orderID, err)
}
// Context carries the deadline — if the caller cancels,
// the payment call respects it.
return s.payments.Charge(ctx, order.CustomerID, order.Total)
}
# Python does not have context.Context natively, but the pattern
# applies through asyncio task cancellation and contextvars.
import contextvars
request_id: contextvars.ContextVar[str] = contextvars.ContextVar("request_id")
async def process_order(order_id: str) -> None:
# contextvars propagate automatically through async calls.
rid = request_id.get()
logger.info("processing order", order_id=order_id, request_id=rid)
order = await orders.get_by_id(order_id)
await payments.charge(order.customer_id, order.total)
3. Functional Options Pattern Link to heading
Constructors with many parameters become unreadable. Functional options provide a clean API for optional configuration without breaking existing callers when new options are added. Each option is a function that modifies a config struct.
type Server struct {
addr string
readTimeout time.Duration
writeTimeout time.Duration
logger Logger
}
type Option func(*Server)
func WithReadTimeout(d time.Duration) Option {
return func(s *Server) { s.readTimeout = d }
}
func WithLogger(l Logger) Option {
return func(s *Server) { s.logger = l }
}
// NewServer applies defaults, then overrides with caller options.
func NewServer(addr string, opts ...Option) *Server {
s := &Server{
addr: addr,
readTimeout: 5 * time.Second,
writeTimeout: 10 * time.Second,
logger: noopLogger{},
}
for _, opt := range opts {
opt(s)
}
return s
}
// Usage: clean, self-documenting, extensible.
srv := NewServer(":8080",
WithReadTimeout(10*time.Second),
WithLogger(logger),
)
# Python achieves similar ergonomics with dataclass defaults
# and keyword-only arguments.
from dataclasses import dataclass, field
@dataclass(frozen=True)
class ServerConfig:
addr: str
read_timeout: float = 5.0
write_timeout: float = 10.0
logger: Logger = field(default_factory=NoopLogger)
class Server:
def __init__(self, addr: str, **kwargs) -> None:
self._config = ServerConfig(addr=addr, **kwargs)
4. Explicit Dependency Injection Without Frameworks Link to heading
Pass dependencies through constructors. No global variables, no service locators, no annotation-driven magic. The dependency graph is visible in the composition root — the single place where everything is wired together.
func main() {
// Composition root: every dependency is explicit.
db := postgres.NewClient(cfg.DatabaseURL)
cache := redis.NewClient(cfg.RedisURL)
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
userRepo := postgres.NewUserRepository(db)
userCache := redis.NewUserCache(cache)
userService := userservice.New(userRepo, userCache, logger)
userHandler := api.NewUserHandler(userService)
mux := http.NewServeMux()
mux.Handle("/users", userHandler)
http.ListenAndServe(":8080", mux)
}
# Python equivalent — no frameworks, just constructors.
def create_app(settings: Settings) -> FastAPI:
db = AsyncpgClient(settings.database_url)
cache = RedisClient(settings.redis_url)
logger = structlog.get_logger()
user_repo = PostgresUserRepository(db)
user_service = UserService(repo=user_repo, cache=cache, logger=logger)
user_controller = UserController(service=user_service)
app = FastAPI()
app.include_router(user_controller.router)
return app
5. Composition Over Inheritance Link to heading
Go has no inheritance. Python has it but should avoid it for most structural patterns. Embed structs in Go. Compose objects in Python. Inheritance creates rigid hierarchies; composition creates flexible assemblies.
// Composition via embedding — Logger gains both capabilities
// without inheriting from either.
type TimestampWriter struct{ w io.Writer }
func (tw *TimestampWriter) Write(p []byte) (int, error) {
prefix := []byte(time.Now().Format(time.RFC3339) + " ")
return tw.w.Write(append(prefix, p...))
}
type PrefixWriter struct {
Prefix string
w io.Writer
}
func (pw *PrefixWriter) Write(p []byte) (int, error) {
return pw.w.Write(append([]byte(pw.Prefix), p...))
}
// Compose behaviors by wrapping.
out := &TimestampWriter{w: &PrefixWriter{Prefix: "[APP] ", w: os.Stdout}}
# Python: compose via delegation, not inheritance.
class RetryingClient:
"""Adds retry behavior to any HTTP client."""
def __init__(self, inner: HttpClient, max_retries: int = 3) -> None:
self._inner = inner
self._max_retries = max_retries
async def get(self, url: str) -> Response:
for attempt in range(self._max_retries):
try:
return await self._inner.get(url)
except TransientError:
if attempt == self._max_retries - 1:
raise
await asyncio.sleep(2 ** attempt)
6. Compile-Time Interface Satisfaction Assertions Link to heading
Catch missing interface methods at compile time, not at runtime. A single line guarantees that a type satisfies an interface, and the compiler enforces it on every build.
// If UserRepository is missing any method from Repository[User, UserParams, string],
// this line fails compilation — not a test, not a runtime panic.
var _ Repository[User, UserParams, string] = (*UserRepository)(nil)
type UserRepository struct {
db *sql.DB
}
func (r *UserRepository) GetByID(ctx context.Context, id string) (User, error) {
// implementation
}
func (r *UserRepository) Create(ctx context.Context, u User) (User, error) {
// implementation
}
# Python: use runtime_checkable Protocol + assert at module level.
from typing import Protocol, runtime_checkable
@runtime_checkable
class Repository(Protocol):
def get_by_id(self, entity_id: str) -> object | None: ...
class UserRepository:
def get_by_id(self, entity_id: str) -> User | None:
...
# Module-level assertion — fails on import if contract is broken.
assert isinstance(UserRepository(), Repository)
7. Interface Segregation Link to heading
Split fat interfaces into focused ones. A consumer that only writes should not depend on an interface that also reads. Segregation reduces coupling and makes testing straightforward — fewer methods to fake.
// Bad: one fat interface forces every consumer to depend on everything.
type UserStore interface {
Create(ctx context.Context, u User) error
GetByID(ctx context.Context, id string) (User, error)
Update(ctx context.Context, u User) error
Delete(ctx context.Context, id string) error
List(ctx context.Context, params ListParams) ([]User, error)
Count(ctx context.Context) (int64, error)
}
// Good: segregated interfaces — consumers depend only on what they use.
type UserReader interface {
GetByID(ctx context.Context, id string) (User, error)
List(ctx context.Context, params ListParams) ([]User, error)
}
type UserWriter interface {
Create(ctx context.Context, u User) error
Update(ctx context.Context, u User) error
}
// A single implementation can satisfy both.
type userRepo struct{ db *sql.DB }
var _ UserReader = (*userRepo)(nil)
var _ UserWriter = (*userRepo)(nil)
8. Factory Constructor Pattern (NewX) Link to heading
Go constructors are plain functions named NewX. They validate inputs, set defaults, and return concrete types. Python uses __init__ with the same discipline — validate early, fail loud.
type Client struct {
baseURL string
httpClient *http.Client
logger Logger
}
// New validates and applies defaults. Callers cannot create
// a half-initialized Client by using a struct literal.
func New(baseURL string, opts ...Option) (*Client, error) {
if baseURL == "" {
return nil, errors.New("base URL is required")
}
c := &Client{
baseURL: strings.TrimRight(baseURL, "/"),
httpClient: &http.Client{Timeout: 30 * time.Second},
logger: noopLogger{},
}
for _, opt := range opts {
opt(c)
}
return c, nil
}
9. Interface Nil-Safety Patterns Link to heading
A nil pointer behind an interface is not a nil interface — it passes nil checks but panics on method calls. Guard against this with constructor validation and typed nil checks.
type Logger interface {
Info(msg string, args ...any)
}
func NewService(logger Logger) (*Service, error) {
// This catches the case where a caller passes (*ZapLogger)(nil),
// which is a non-nil interface wrapping a nil pointer.
if logger == nil {
return nil, errors.New("logger is required")
}
return &Service{logger: logger}, nil
}
// For optional dependencies, use a no-op implementation — never nil.
func NewServiceWithDefaults(logger Logger) *Service {
if logger == nil {
logger = noopLogger{}
}
return &Service{logger: logger}
}
Part II: Configuration and Initialization Link to heading
10. Immutable Configuration Objects Link to heading
Configuration should be loaded once, validated, and then frozen. Mutable configuration creates race conditions and makes it impossible to reason about what values a running service is using.
// Config is a value type — no pointers, no mutation after construction.
type Config struct {
DatabaseURL string
RedisURL string
Port int
LogLevel string
}
func LoadConfig() (Config, error) {
cfg := Config{
DatabaseURL: os.Getenv("DATABASE_URL"),
RedisURL: os.Getenv("REDIS_URL"),
Port: getEnvInt("PORT", 8080),
LogLevel: getEnvOrDefault("LOG_LEVEL", "info"),
}
if cfg.DatabaseURL == "" {
return Config{}, errors.New("DATABASE_URL is required")
}
return cfg, nil // returned by value — caller gets a copy
}
# Python: frozen dataclass or Pydantic BaseSettings.
from pydantic_settings import BaseSettings
class Settings(BaseSettings):
database_url: str
redis_url: str
port: int = 8080
log_level: str = "info"
model_config = {"frozen": True} # immutable after creation
11. Builder-Style Configuration APIs Link to heading
When constructing complex objects with many optional fields, a builder provides a fluent, readable API that guides the caller through valid configurations.
type Pipeline struct {
stages []Stage
retries int
timeout time.Duration
logger Logger
}
type PipelineBuilder struct {
p Pipeline
}
func NewPipelineBuilder() *PipelineBuilder {
return &PipelineBuilder{p: Pipeline{retries: 3, timeout: 30 * time.Second}}
}
func (b *PipelineBuilder) WithStage(s Stage) *PipelineBuilder {
b.p.stages = append(b.p.stages, s)
return b
}
func (b *PipelineBuilder) WithRetries(n int) *PipelineBuilder {
b.p.retries = n
return b
}
func (b *PipelineBuilder) Build() (*Pipeline, error) {
if len(b.p.stages) == 0 {
return nil, errors.New("pipeline requires at least one stage")
}
return &b.p, nil
}
// Usage reads like a specification.
pipeline, err := NewPipelineBuilder().
WithStage(validateStage).
WithStage(transformStage).
WithStage(publishStage).
WithRetries(5).
Build()
12. Lazy Initialization with sync.Once Link to heading
Defer expensive initialization until first use, but guarantee it runs exactly once — even under concurrent access. sync.Once is the idiomatic tool for this in Go.
type TemplateCache struct {
once sync.Once
templates *template.Template
err error
}
func (tc *TemplateCache) Get() (*template.Template, error) {
tc.once.Do(func() {
tc.templates, tc.err = template.ParseGlob("templates/*.html")
})
return tc.templates, tc.err
}
# Python: functools.cache or a manual flag.
import functools
@functools.cache
def get_db_engine() -> Engine:
"""Created on first call, reused on subsequent calls."""
return create_engine(settings.database_url)
13. Declarative Configuration Loaders Link to heading
Load configuration from multiple sources — environment variables, files, defaults — with a single declarative definition. The loader handles precedence: environment overrides file, file overrides default.
type Config struct {
Port int `env:"PORT" default:"8080" validate:"min=1,max=65535"`
LogLevel string `env:"LOG_LEVEL" default:"info" validate:"oneof=debug info warn error"`
DBHost string `env:"DB_HOST" required:"true"`
}
func Load() (Config, error) {
var cfg Config
if err := envconfig.Process("APP", &cfg); err != nil {
return Config{}, fmt.Errorf("load config: %w", err)
}
if err := validator.New().Struct(cfg); err != nil {
return Config{}, fmt.Errorf("validate config: %w", err)
}
return cfg, nil
}
14. Validation-Before-Startup Link to heading
Validate all configuration and connectivity before the service begins accepting traffic. A service that starts successfully should be fully operational — not half-initialized and failing on the first request.
func main() {
cfg, err := config.Load()
if err != nil {
log.Fatalf("config: %v", err)
}
// Validate connectivity before serving.
db, err := sql.Open("postgres", cfg.DatabaseURL)
if err != nil {
log.Fatalf("db connect: %v", err)
}
if err := db.PingContext(context.Background()); err != nil {
log.Fatalf("db ping: %v", err)
}
cache, err := redis.Dial(cfg.RedisURL)
if err != nil {
log.Fatalf("redis: %v", err)
}
log.Println("all dependencies healthy, starting server")
// ... start HTTP server
}
15. Explicit Startup Dependency Ordering Link to heading
Services often have ordering requirements — the database must connect before repositories initialize, the cache must be ready before the service layer starts. Make the ordering explicit in the composition root rather than relying on implicit timing.
func bootstrap(ctx context.Context, cfg Config) (*App, error) {
// Phase 1: Infrastructure (no dependencies)
logger := logger.New(cfg.LogLevel)
metrics := metrics.New(cfg.MetricsAddr)
// Phase 2: Clients (depend on infrastructure)
db, err := postgres.Connect(ctx, cfg.DatabaseURL)
if err != nil {
return nil, fmt.Errorf("database: %w", err)
}
// Phase 3: Repositories (depend on clients)
userRepo := postgres.NewUserRepo(db)
// Phase 4: Services (depend on repositories)
userSvc := userservice.New(userRepo, logger)
// Phase 5: Transport (depend on services)
handler := api.NewHandler(userSvc, logger, metrics)
return &App{handler: handler, db: db}, nil
}
16. Declarative Registration / Init Patterns Link to heading
Use init-time registration for plugins, codecs, or drivers — but keep the registry explicit and testable. Avoid init() functions that modify global state invisibly.
// Registry is explicit — no hidden init() side effects.
type CodecRegistry struct {
codecs map[string]Codec
}
func NewCodecRegistry() *CodecRegistry {
return &CodecRegistry{codecs: make(map[string]Codec)}
}
func (r *CodecRegistry) Register(name string, c Codec) {
r.codecs[name] = c
}
func (r *CodecRegistry) Get(name string) (Codec, error) {
c, ok := r.codecs[name]
if !ok {
return nil, fmt.Errorf("unknown codec: %s", name)
}
return c, nil
}
// Registration happens in the composition root, not in init().
registry := NewCodecRegistry()
registry.Register("json", jsonCodec{})
registry.Register("proto", protoCodec{})
Part III: Error Handling Link to heading
17. Error Wrapping with Contextual Errors Link to heading
Wrap errors with fmt.Errorf("%w") to add context at each call site while preserving the original error for inspection. The resulting error chain tells a complete story: what failed, where, and why.
func (s *Service) GetUser(ctx context.Context, id string) (User, error) {
user, err := s.repo.GetByID(ctx, id)
if err != nil {
// Adds context ("get user") while preserving the original error.
// Callers can still use errors.Is/As to inspect the root cause.
return User{}, fmt.Errorf("get user %s: %w", id, err)
}
return user, nil
}
// Produces error chains like:
// "get user abc123: query users table: connection refused"
# Python: chain exceptions with `from`.
class UserService:
async def get_user(self, user_id: str) -> User:
try:
return await self._repo.get_by_id(user_id)
except DatabaseError as exc:
raise UserNotFoundError(
f"get user {user_id}"
) from exc # preserves the original traceback
18. Sentinel Errors and Typed Errors Link to heading
Define sentinel errors for expected conditions and typed errors for conditions that carry data. Use errors.Is to check sentinels and errors.As to extract typed errors.
// Sentinel errors — expected conditions, not bugs.
var (
ErrNotFound = errors.New("not found")
ErrConflict = errors.New("conflict")
ErrUnauthorized = errors.New("unauthorized")
)
// Typed error — carries structured context.
type ValidationError struct {
Field string
Message string
}
func (e *ValidationError) Error() string {
return fmt.Sprintf("validation: %s: %s", e.Field, e.Message)
}
// Checking errors at the call site.
user, err := svc.GetUser(ctx, id)
if errors.Is(err, ErrNotFound) {
http.Error(w, "user not found", http.StatusNotFound)
return
}
var valErr *ValidationError
if errors.As(err, &valErr) {
http.Error(w, valErr.Message, http.StatusBadRequest)
return
}
19. Structured Error Classification Link to heading
Classify errors into categories that drive retry behavior and HTTP status codes. A single classification function replaces scattered switch statements.
type ErrorKind int
const (
KindTransient ErrorKind = iota // retry
KindNotFound // 404
KindValidation // 400
KindUnauthorized // 401
KindInternal // 500
)
func Classify(err error) ErrorKind {
if errors.Is(err, ErrNotFound) {
return KindNotFound
}
var valErr *ValidationError
if errors.As(err, &valErr) {
return KindValidation
}
if isTransient(err) {
return KindTransient
}
return KindInternal
}
func HTTPStatus(kind ErrorKind) int {
switch kind {
case KindNotFound: return http.StatusNotFound
case KindValidation: return http.StatusBadRequest
case KindUnauthorized: return http.StatusUnauthorized
case KindTransient: return http.StatusServiceUnavailable
default: return http.StatusInternalServerError
}
}
20. Retryable vs Terminal Error Distinction Link to heading
Not all errors deserve a retry. Network timeouts are retryable. Validation failures are terminal. Make this distinction explicit so retry logic does not waste time on errors that will never succeed.
type RetryableError struct {
Err error
}
func (e *RetryableError) Error() string { return e.Err.Error() }
func (e *RetryableError) Unwrap() error { return e.Err }
func IsRetryable(err error) bool {
var retryable *RetryableError
return errors.As(err, &retryable)
}
// In the repository layer — classify at the source.
func (r *repo) Query(ctx context.Context, q string) ([]Row, error) {
rows, err := r.db.QueryContext(ctx, q)
if err != nil {
if isConnectionError(err) {
return nil, &RetryableError{Err: fmt.Errorf("query: %w", err)}
}
return nil, fmt.Errorf("query: %w", err) // terminal
}
return scan(rows)
}
21. Error Aggregation / Multi-Error Handling Link to heading
When validating multiple fields or running parallel operations, collect all errors rather than failing on the first. Go 1.20+ provides errors.Join; Python uses exception groups.
func Validate(order Order) error {
var errs []error
if order.CustomerID == "" {
errs = append(errs, errors.New("customer_id is required"))
}
if order.Total <= 0 {
errs = append(errs, errors.New("total must be positive"))
}
if len(order.Items) == 0 {
errs = append(errs, errors.New("at least one item is required"))
}
return errors.Join(errs...) // nil if no errors
}
# Python 3.11+ ExceptionGroup for parallel error collection.
async def validate_all(items: list[Item]) -> None:
errors = []
for item in items:
try:
validate_item(item)
except ValidationError as e:
errors.append(e)
if errors:
raise ExceptionGroup("validation failed", errors)
22. Panic Recovery at Service Boundaries Link to heading
Panics in Go should not crash the entire server. Recover at the outermost boundary — the HTTP middleware — log the stack trace, and return a 500. Never recover inside business logic; panics there indicate bugs that should be fixed, not hidden.
func RecoveryMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
defer func() {
if rec := recover(); rec != nil {
stack := debug.Stack()
slog.Error("panic recovered",
"error", rec,
"stack", string(stack),
"path", r.URL.Path,
)
http.Error(w, "internal server error", http.StatusInternalServerError)
}
}()
next.ServeHTTP(w, r)
})
}
Part IV: Context and Resource Management Link to heading
23. Timeout-Wrapped External Calls Link to heading
Every call to an external system needs a timeout. Without one, a hung upstream holds a goroutine and a connection indefinitely. Wrap external calls with a derived context that enforces a deadline.
func (c *Client) FetchUser(ctx context.Context, id string) (User, error) {
// 5-second timeout on this specific call, regardless of parent deadline.
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet,
fmt.Sprintf("%s/users/%s", c.baseURL, id), nil)
if err != nil {
return User{}, fmt.Errorf("build request: %w", err)
}
resp, err := c.httpClient.Do(req)
if err != nil {
return User{}, fmt.Errorf("fetch user %s: %w", id, err)
}
defer resp.Body.Close()
var user User
if err := json.NewDecoder(resp.Body).Decode(&user); err != nil {
return User{}, fmt.Errorf("decode user: %w", err)
}
return user, nil
}
# Python: asyncio.timeout (3.11+) or async_timeout.
async def fetch_user(self, user_id: str) -> User:
async with asyncio.timeout(5.0):
resp = await self._session.get(f"/users/{user_id}")
resp.raise_for_status()
return User(**await resp.json())
24. Resource Cleanup with defer Link to heading
defer in Go guarantees cleanup runs when the function exits — whether normally or via panic. Use it for closing files, releasing locks, stopping timers, and rolling back transactions.
func (s *Service) ProcessFile(ctx context.Context, path string) error {
f, err := os.Open(path)
if err != nil {
return fmt.Errorf("open %s: %w", path, err)
}
defer f.Close() // guaranteed cleanup
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return fmt.Errorf("begin tx: %w", err)
}
defer tx.Rollback() // no-op if already committed
// ... process file, insert records ...
return tx.Commit() // if this succeeds, deferred Rollback is a no-op
}
# Python: context managers serve the same role.
async def process_file(self, path: str) -> None:
async with aiofiles.open(path) as f:
async with self._db.transaction() as tx:
data = await f.read()
await tx.execute("INSERT INTO ...", data)
# Transaction commits on clean exit, rolls back on exception.
25. Context-Aware Retries and Cancellation Link to heading
Retry loops must check context cancellation between attempts. A retry that ignores context will continue hammering a failing upstream even after the caller has given up.
func RetryWithContext(ctx context.Context, maxAttempts int, fn func() error) error {
var lastErr error
for attempt := range maxAttempts {
if err := ctx.Err(); err != nil {
return fmt.Errorf("retry cancelled after %d attempts: %w", attempt, err)
}
lastErr = fn()
if lastErr == nil {
return nil
}
if !IsRetryable(lastErr) {
return lastErr // terminal error — do not retry
}
backoff := time.Duration(1<<attempt) * 100 * time.Millisecond
select {
case <-time.After(backoff):
case <-ctx.Done():
return ctx.Err()
}
}
return fmt.Errorf("exhausted %d attempts: %w", maxAttempts, lastErr)
}
Part V: Concurrency Patterns Link to heading
26. Worker Pool Pattern Link to heading
Bound the number of concurrent operations. Unbounded goroutine creation leads to memory exhaustion and thundering-herd problems. A worker pool with a fixed number of goroutines pulling from a channel is the standard solution.
func WorkerPool[T any](ctx context.Context, workers int, jobs <-chan T, process func(T) error) error {
g, ctx := errgroup.WithContext(ctx)
for range workers {
g.Go(func() error {
for {
select {
case job, ok := <-jobs:
if !ok {
return nil
}
if err := process(job); err != nil {
slog.Error("job failed", "error", err)
}
case <-ctx.Done():
return ctx.Err()
}
}
})
}
return g.Wait()
}
# Python: asyncio.TaskGroup with a semaphore for bounded concurrency.
async def worker_pool(
items: list[T],
process: Callable[[T], Awaitable[None]],
max_workers: int = 10,
) -> None:
sem = asyncio.Semaphore(max_workers)
async def bounded(item: T) -> None:
async with sem:
await process(item)
async with asyncio.TaskGroup() as tg:
for item in items:
tg.create_task(bounded(item))
27. Channel-Based Coordination Link to heading
Channels are Go’s primitive for communication between goroutines. Prefer channels over shared memory and mutexes when goroutines need to exchange data or signal completion.
// done channel signals all workers to stop.
func produce(ctx context.Context, out chan<- Event) {
defer close(out) // closing signals consumers that no more data is coming
for {
event, err := pollSource(ctx)
if err != nil {
return
}
select {
case out <- event:
case <-ctx.Done():
return
}
}
}
func consume(ctx context.Context, in <-chan Event) {
for {
select {
case event, ok := <-in:
if !ok {
return // channel closed — producer is done
}
handleEvent(event)
case <-ctx.Done():
return
}
}
}
28. Fan-Out / Fan-In Concurrency Pipelines Link to heading
Fan-out distributes work across multiple goroutines. Fan-in merges results back into a single channel. The pattern enables parallel processing of independent tasks with a single collection point.
func FanOutFanIn[In, Out any](
ctx context.Context,
inputs []In,
workers int,
process func(context.Context, In) (Out, error),
) ([]Out, error) {
in := make(chan In)
out := make(chan Out, len(inputs))
// Fan-out: distribute inputs to workers.
g, ctx := errgroup.WithContext(ctx)
for range workers {
g.Go(func() error {
for input := range in {
result, err := process(ctx, input)
if err != nil {
return err
}
out <- result
}
return nil
})
}
// Feed inputs.
go func() {
defer close(in)
for _, input := range inputs {
select {
case in <- input:
case <-ctx.Done():
return
}
}
}()
// Fan-in: collect results after all workers finish.
err := g.Wait()
close(out)
if err != nil {
return nil, err
}
var results []Out
for result := range out {
results = append(results, result)
}
return results, nil
}
29. Rate-Limited Work Queues Link to heading
When calling external services, rate limiting prevents overwhelming the upstream. A token-bucket or ticker-based approach controls throughput at the source.
func RateLimitedProcessor(ctx context.Context, jobs <-chan Job, rps int) error {
limiter := rate.NewLimiter(rate.Limit(rps), 1) // golang.org/x/time/rate
for {
select {
case job, ok := <-jobs:
if !ok {
return nil
}
// Wait blocks until a token is available or ctx is cancelled.
if err := limiter.Wait(ctx); err != nil {
return err
}
if err := process(job); err != nil {
slog.Error("job failed", "error", err)
}
case <-ctx.Done():
return ctx.Err()
}
}
}
30. Bounded Concurrency with Semaphores Link to heading
A semaphore limits the number of goroutines executing a critical section. Unlike a worker pool that creates fixed goroutines, a semaphore allows dynamic goroutine creation while capping concurrent access.
func ProcessAll(ctx context.Context, items []Item) error {
sem := make(chan struct{}, 10) // at most 10 concurrent operations
g, ctx := errgroup.WithContext(ctx)
for _, item := range items {
g.Go(func() error {
sem <- struct{}{} // acquire
defer func() { <-sem }() // release
return processItem(ctx, item)
})
}
return g.Wait()
}
31. Ownership-Based Concurrency Design Link to heading
Each piece of mutable state should be owned by exactly one goroutine. Other goroutines interact with that state by sending messages to the owner. This eliminates data races without mutexes.
// The stats goroutine owns the counters map. No mutex needed.
type StatsCollector struct {
updates chan stat
queries chan statsQuery
}
type stat struct {
key string
delta int64
}
type statsQuery struct {
reply chan map[string]int64
}
func (sc *StatsCollector) Run(ctx context.Context) {
counters := make(map[string]int64)
for {
select {
case s := <-sc.updates:
counters[s.key] += s.delta
case q := <-sc.queries:
snapshot := make(map[string]int64, len(counters))
for k, v := range counters {
snapshot[k] = v
}
q.reply <- snapshot
case <-ctx.Done():
return
}
}
}
func (sc *StatsCollector) Increment(key string) {
sc.updates <- stat{key: key, delta: 1}
}
32. Single-Writer Concurrency Pattern Link to heading
Multiple readers are safe. Multiple writers cause contention. Design data flows so that exactly one goroutine writes to any given data structure. Readers access snapshots or immutable copies.
type ConfigWatcher struct {
mu sync.RWMutex
config atomic.Pointer[Config] // single writer, multiple readers
}
// Update is called by exactly one goroutine (the config watcher).
func (cw *ConfigWatcher) Update(cfg Config) {
cw.config.Store(&cfg)
}
// Get is called by any number of goroutines concurrently — lock-free.
func (cw *ConfigWatcher) Get() Config {
return *cw.config.Load()
}
33. Actor-Like Goroutine Ownership Model Link to heading
Each “actor” goroutine manages its own state and communicates exclusively through channels. This maps naturally to Go’s concurrency model and eliminates shared-state bugs.
type OrderProcessor struct {
orders chan Order
done chan struct{}
}
func NewOrderProcessor() *OrderProcessor {
op := &OrderProcessor{
orders: make(chan Order, 100),
done: make(chan struct{}),
}
go op.run()
return op
}
func (op *OrderProcessor) run() {
defer close(op.done)
pending := make(map[string]*Order)
for order := range op.orders {
// All state mutation happens here — single goroutine, no races.
if existing, ok := pending[order.ID]; ok {
existing.Merge(order)
} else {
pending[order.ID] = &order
}
if order.IsComplete() {
delete(pending, order.ID)
// publish completion event
}
}
}
func (op *OrderProcessor) Submit(o Order) { op.orders <- o }
func (op *OrderProcessor) Stop() { close(op.orders); <-op.done }
34. Goroutine Lifecycle Ownership Rules Link to heading
The goroutine that starts a goroutine is responsible for ensuring it stops. Leaked goroutines are memory leaks. Use errgroup, context cancellation, or explicit shutdown channels to enforce lifecycle ownership.
type Server struct {
cancel context.CancelFunc
g *errgroup.Group
}
func NewServer(ctx context.Context) *Server {
ctx, cancel := context.WithCancel(ctx)
g, ctx := errgroup.WithContext(ctx)
s := &Server{cancel: cancel, g: g}
// The server owns these goroutines — it starts them, it stops them.
g.Go(func() error { return s.runHTTP(ctx) })
g.Go(func() error { return s.runGRPC(ctx) })
g.Go(func() error { return s.runMetrics(ctx) })
return s
}
func (s *Server) Shutdown() error {
s.cancel() // signal all owned goroutines to stop
return s.g.Wait() // wait for all to finish
}
Part VI: Lifecycle and Orchestration Link to heading
35. Reconciliation Loop Pattern Link to heading
Instead of reacting to individual events, periodically compare desired state with actual state and take corrective action. This pattern is self-healing — if an event is missed or a handler crashes, the next reconciliation cycle fixes the drift.
func (c *Controller) reconcileLoop(ctx context.Context) error {
ticker := time.NewTicker(30 * time.Second)
defer ticker.Stop()
for {
select {
case <-ticker.C:
if err := c.reconcile(ctx); err != nil {
slog.Error("reconciliation failed", "error", err)
}
case <-ctx.Done():
return ctx.Err()
}
}
}
func (c *Controller) reconcile(ctx context.Context) error {
desired, err := c.store.ListDesired(ctx)
if err != nil {
return fmt.Errorf("list desired: %w", err)
}
actual, err := c.store.ListActual(ctx)
if err != nil {
return fmt.Errorf("list actual: %w", err)
}
for _, d := range desired {
a, exists := actual[d.ID]
if !exists {
if err := c.create(ctx, d); err != nil {
return err
}
} else if !d.Equal(a) {
if err := c.update(ctx, d); err != nil {
return err
}
}
}
return nil
}
36. Graceful Shutdown with Contexts and Signals Link to heading
Trap OS signals, cancel the root context, drain in-flight work, and close resources in reverse order of initialization. A service that exits cleanly under SIGTERM is a service that can be deployed without request drops.
func main() {
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
app, err := bootstrap(ctx, loadConfig())
if err != nil {
log.Fatalf("bootstrap: %v", err)
}
// Start server in a goroutine.
srv := &http.Server{Addr: ":8080", Handler: app.handler}
go func() {
if err := srv.ListenAndServe(); err != http.ErrServerClosed {
log.Fatalf("serve: %v", err)
}
}()
<-ctx.Done() // block until signal received
slog.Info("shutting down")
// Give in-flight requests 15 seconds to complete.
shutdownCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
if err := srv.Shutdown(shutdownCtx); err != nil {
slog.Error("shutdown error", "error", err)
}
app.db.Close()
slog.Info("shutdown complete")
}
37. Manager/Controller Lifecycle Orchestration Link to heading
A manager coordinates the lifecycle of multiple subsystems — starting them in order, monitoring health, and shutting them down cleanly on exit.
type Manager struct {
services []Service
}
type Service interface {
Name() string
Start(ctx context.Context) error
Stop(ctx context.Context) error
Healthy() bool
}
func (m *Manager) Start(ctx context.Context) error {
for _, svc := range m.services {
slog.Info("starting", "service", svc.Name())
if err := svc.Start(ctx); err != nil {
// Rollback: stop already-started services in reverse.
m.stopReverse(ctx, m.services[:len(m.services)-1])
return fmt.Errorf("start %s: %w", svc.Name(), err)
}
}
return nil
}
func (m *Manager) Stop(ctx context.Context) {
// Stop in reverse order of startup.
for i := len(m.services) - 1; i >= 0; i-- {
slog.Info("stopping", "service", m.services[i].Name())
if err := m.services[i].Stop(ctx); err != nil {
slog.Error("stop error", "service", m.services[i].Name(), "error", err)
}
}
}
38. Background Daemon / Service Manager Pattern Link to heading
Long-running background tasks — cache warming, metric flushing, config reloading — need the same lifecycle management as the primary server. Run them as managed goroutines that respect context cancellation.
type Daemon struct {
name string
interval time.Duration
fn func(context.Context) error
}
func (d *Daemon) Run(ctx context.Context) error {
slog.Info("daemon started", "name", d.name, "interval", d.interval)
ticker := time.NewTicker(d.interval)
defer ticker.Stop()
for {
select {
case <-ticker.C:
if err := d.fn(ctx); err != nil {
slog.Error("daemon tick failed", "name", d.name, "error", err)
}
case <-ctx.Done():
slog.Info("daemon stopped", "name", d.name)
return nil
}
}
}
// Usage: managed alongside the HTTP server.
g, ctx := errgroup.WithContext(ctx)
g.Go(func() error { return httpServer.Serve(ctx) })
g.Go(func() error { return cacheWarmer.Run(ctx) })
g.Go(func() error { return metricsFlush.Run(ctx) })
return g.Wait()
39. Plugin Registry Pattern Link to heading
Allow components to register themselves with a central registry. The registry provides discovery and lifecycle management without tight coupling between the registrar and the plugins.
type HealthChecker interface {
Name() string
Check(ctx context.Context) error
}
type HealthRegistry struct {
mu sync.RWMutex
checkers []HealthChecker
}
func (r *HealthRegistry) Register(c HealthChecker) {
r.mu.Lock()
defer r.mu.Unlock()
r.checkers = append(r.checkers, c)
}
func (r *HealthRegistry) CheckAll(ctx context.Context) map[string]error {
r.mu.RLock()
defer r.mu.RUnlock()
results := make(map[string]error, len(r.checkers))
for _, c := range r.checkers {
results[c.Name()] = c.Check(ctx)
}
return results
}
Part VII: Resilience and Reliability Link to heading
40. Exponential Backoff and Retry Link to heading
Retry transient failures with exponential backoff and jitter. Without jitter, retrying clients synchronize and create thundering-herd spikes. Without backoff, aggressive retries overwhelm an already struggling upstream.
func Retry(ctx context.Context, maxAttempts int, fn func() error) error {
for attempt := range maxAttempts {
err := fn()
if err == nil {
return nil
}
if !IsRetryable(err) {
return err
}
if attempt == maxAttempts-1 {
return fmt.Errorf("exhausted %d attempts: %w", maxAttempts, err)
}
// Exponential backoff with jitter.
base := time.Duration(1<<attempt) * 100 * time.Millisecond
jitter := time.Duration(rand.Int64N(int64(base / 2)))
backoff := base + jitter
select {
case <-time.After(backoff):
case <-ctx.Done():
return ctx.Err()
}
}
return nil
}
import random
async def retry(
fn: Callable[[], Awaitable[T]],
max_attempts: int = 3,
base_delay: float = 0.1,
) -> T:
for attempt in range(max_attempts):
try:
return await fn()
except RetryableError:
if attempt == max_attempts - 1:
raise
delay = base_delay * (2 ** attempt) + random.uniform(0, base_delay)
await asyncio.sleep(delay)
41. Circuit Breaker Pattern Link to heading
A circuit breaker stops calling a failing upstream after a threshold of consecutive failures. It transitions through closed (normal), open (failing fast), and half-open (testing recovery) states. This prevents a degraded dependency from consuming resources and cascading failures.
type CircuitBreaker struct {
mu sync.Mutex
failures int
threshold int
state State
lastFailure time.Time
cooldown time.Duration
}
func (cb *CircuitBreaker) Execute(fn func() error) error {
cb.mu.Lock()
if cb.state == Open {
if time.Since(cb.lastFailure) < cb.cooldown {
cb.mu.Unlock()
return ErrCircuitOpen
}
cb.state = HalfOpen // allow one probe request
}
cb.mu.Unlock()
err := fn()
cb.mu.Lock()
defer cb.mu.Unlock()
if err != nil {
cb.failures++
cb.lastFailure = time.Now()
if cb.failures >= cb.threshold {
cb.state = Open
}
return err
}
cb.failures = 0
cb.state = Closed
return nil
}
42. Bulkhead Isolation Pattern Link to heading
Isolate different categories of work so that a failure in one does not exhaust resources shared by others. A bulkhead for each upstream service prevents a slow dependency from starving unrelated request paths.
type Bulkhead struct {
sem chan struct{}
}
func NewBulkhead(maxConcurrent int) *Bulkhead {
return &Bulkhead{sem: make(chan struct{}, maxConcurrent)}
}
func (b *Bulkhead) Execute(ctx context.Context, fn func() error) error {
select {
case b.sem <- struct{}{}:
defer func() { <-b.sem }()
return fn()
case <-ctx.Done():
return ctx.Err()
default:
return ErrBulkheadFull // fail fast if all slots are taken
}
}
// Usage: separate bulkheads per upstream.
var (
paymentBulkhead = NewBulkhead(20)
inventoryBulkhead = NewBulkhead(50)
)
43. Idempotent Handlers and Reconciliation Link to heading
Every operation that can be retried must be idempotent — executing it twice produces the same result as executing it once. Use idempotency keys to deduplicate requests and store the result of the first execution.
func (h *Handler) CreateOrder(ctx context.Context, req CreateOrderRequest) (Order, error) {
// Check if this idempotency key has already been processed.
existing, err := h.idempotencyStore.Get(ctx, req.IdempotencyKey)
if err == nil {
return existing, nil // already processed — return cached result
}
order, err := h.service.CreateOrder(ctx, req)
if err != nil {
return Order{}, err
}
// Store result for deduplication.
_ = h.idempotencyStore.Set(ctx, req.IdempotencyKey, order, 24*time.Hour)
return order, nil
}
44. Retry Transport Wrappers Link to heading
Wrap http.RoundTripper to add retry logic transparently. Every HTTP client in the application benefits without modifying call sites.
type RetryTransport struct {
inner http.RoundTripper
maxAttempts int
}
func (t *RetryTransport) RoundTrip(req *http.Request) (*http.Response, error) {
var lastErr error
for attempt := range t.maxAttempts {
resp, err := t.inner.RoundTrip(req)
if err == nil && resp.StatusCode < 500 {
return resp, nil
}
if err != nil {
lastErr = err
} else {
lastErr = fmt.Errorf("HTTP %d", resp.StatusCode)
resp.Body.Close()
}
backoff := time.Duration(1<<attempt) * 100 * time.Millisecond
select {
case <-time.After(backoff):
case <-req.Context().Done():
return nil, req.Context().Err()
}
}
return nil, lastErr
}
// Applied once — all requests through this client get retries.
client := &http.Client{
Transport: &RetryTransport{inner: http.DefaultTransport, maxAttempts: 3},
}
45. Shared Transport / Client Reuse Link to heading
Create HTTP clients and database connections once and reuse them. Each http.Client maintains a connection pool; creating new clients per request wastes connections and triggers excessive TCP handshakes.
// Bad: new client per request — leaks connections.
func fetchBad(url string) (*http.Response, error) {
client := &http.Client{} // new pool every call
return client.Get(url)
}
// Good: shared client configured once.
var httpClient = &http.Client{
Timeout: 30 * time.Second,
Transport: &http.Transport{
MaxIdleConns: 100,
MaxIdleConnsPerHost: 10,
IdleConnTimeout: 90 * time.Second,
},
}
func fetchGood(ctx context.Context, url string) (*http.Response, error) {
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
return httpClient.Do(req)
}
46. Outbox Pattern for Reliable Event Publishing Link to heading
Writing a database record and publishing an event are two separate operations that can fail independently. The outbox pattern writes the event to a database table within the same transaction as the business data, then a separate process reads and publishes events. This guarantees at-least-once delivery without distributed transactions.
func (s *Service) CreateOrder(ctx context.Context, order Order) error {
return s.db.WithTx(ctx, func(tx *sql.Tx) error {
// Business operation and event written in the same transaction.
if err := insertOrder(tx, order); err != nil {
return fmt.Errorf("insert order: %w", err)
}
event := OutboxEvent{
AggregateID: order.ID,
EventType: "order.created",
Payload: mustMarshal(order),
CreatedAt: time.Now(),
}
if err := insertOutboxEvent(tx, event); err != nil {
return fmt.Errorf("insert outbox event: %w", err)
}
return nil
})
}
// Separate poller reads outbox and publishes to Kafka.
func (p *OutboxPoller) poll(ctx context.Context) error {
events, err := p.store.GetUnpublished(ctx, 100)
if err != nil {
return err
}
for _, event := range events {
if err := p.publisher.Publish(ctx, event); err != nil {
return err
}
_ = p.store.MarkPublished(ctx, event.ID)
}
return nil
}
47. Lease / Leader-Election Coordination Link to heading
In a multi-replica deployment, certain tasks — cron jobs, reconciliation loops, outbox polling — should run on exactly one replica. Leader election via a distributed lock (database row lock, Redis, etcd) ensures single-execution semantics.
func (w *Worker) RunWithLease(ctx context.Context) error {
ticker := time.NewTicker(10 * time.Second)
defer ticker.Stop()
for {
select {
case <-ticker.C:
// Attempt to acquire lease. Only the leader succeeds.
acquired, err := w.lock.TryAcquire(ctx, "reconciler", 30*time.Second)
if err != nil {
slog.Error("lease error", "error", err)
continue
}
if !acquired {
continue // another replica holds the lease
}
if err := w.reconcile(ctx); err != nil {
slog.Error("reconcile failed", "error", err)
}
_ = w.lock.Release(ctx, "reconciler")
case <-ctx.Done():
return nil
}
}
}
Part VIII: Data Access and Transactions Link to heading
48. Repository Pattern (Lightweight) Link to heading
A repository provides a domain-oriented interface to storage. It accepts and returns domain entities — not database rows, not ORM models. The implementation translates between domain and storage representations.
type UserRepository interface {
GetByID(ctx context.Context, id string) (User, error)
Save(ctx context.Context, user User) error
Delete(ctx context.Context, id string) error
}
type postgresUserRepo struct {
db *sql.DB
}
func (r *postgresUserRepo) GetByID(ctx context.Context, id string) (User, error) {
var row userRow
err := r.db.QueryRowContext(ctx,
"SELECT id, email, name, created_at FROM users WHERE id = $1", id,
).Scan(&row.ID, &row.Email, &row.Name, &row.CreatedAt)
if errors.Is(err, sql.ErrNoRows) {
return User{}, ErrNotFound
}
if err != nil {
return User{}, fmt.Errorf("query user: %w", err)
}
return row.toDomain(), nil
}
49. API Boundary DTO Separation Link to heading
Never expose domain entities directly in API responses. Use separate DTOs (Data Transfer Objects) for serialization. This decouples your internal model from your external contract — you can refactor the domain without breaking API clients.
// Domain entity — internal.
type User struct {
ID string
Email string
PasswordHash string // never expose this
CreatedAt time.Time
}
// API DTO — external contract.
type UserResponse struct {
ID string `json:"id"`
Email string `json:"email"`
CreatedAt string `json:"created_at"`
}
func toUserResponse(u User) UserResponse {
return UserResponse{
ID: u.ID,
Email: u.Email,
CreatedAt: u.CreatedAt.Format(time.RFC3339),
}
}
# Python: Pydantic models as DTOs, separate from domain entities.
from pydantic import BaseModel
# Domain entity.
@dataclass
class User:
id: str
email: str
password_hash: str
created_at: datetime
# API DTO — controls what the client sees.
class UserResponse(BaseModel):
id: str
email: str
created_at: datetime
@classmethod
def from_domain(cls, user: User) -> "UserResponse":
return cls(id=user.id, email=user.email, created_at=user.created_at)
50. Explicit Serialization Models Link to heading
Define separate structs for JSON, YAML, and Protobuf serialization. A single struct with multiple tag sets becomes unreadable and couples unrelated concerns. Separate models make each format’s contract explicit.
// Domain model — no serialization tags.
type Order struct {
ID string
Items []Item
Total Money
Status OrderStatus
CreatedAt time.Time
}
// JSON API model.
type OrderJSON struct {
ID string `json:"id"`
Items []ItemJSON `json:"items"`
Total string `json:"total"` // Money serialized as string
Status string `json:"status"`
CreatedAt string `json:"created_at"`
}
// Database model.
type OrderRow struct {
ID string `db:"id"`
Items []byte `db:"items"` // JSONB
TotalCents int64 `db:"total_cents"` // stored as cents
Status string `db:"status"`
CreatedAt time.Time `db:"created_at"`
}
51. Transaction Boundary Encapsulation Link to heading
Wrap transaction management in a helper that handles begin, commit, and rollback. Business logic should not contain transaction plumbing — it should receive a function-scoped transaction and focus on domain operations.
func (db *DB) WithTx(ctx context.Context, fn func(tx *sql.Tx) error) error {
tx, err := db.BeginTx(ctx, nil)
if err != nil {
return fmt.Errorf("begin tx: %w", err)
}
defer tx.Rollback() // no-op after commit
if err := fn(tx); err != nil {
return err
}
return tx.Commit()
}
// Usage: business logic stays clean.
err := db.WithTx(ctx, func(tx *sql.Tx) error {
if err := createOrder(tx, order); err != nil {
return err
}
return updateInventory(tx, order.Items)
})
52. Optimistic Concurrency / Version Checks Link to heading
Prevent lost updates by including a version or timestamp in update queries. If the version has changed since the read, the update fails — the caller must re-read and retry.
func (r *repo) Update(ctx context.Context, user User) error {
result, err := r.db.ExecContext(ctx,
`UPDATE users SET email = $1, name = $2, version = version + 1
WHERE id = $3 AND version = $4`,
user.Email, user.Name, user.ID, user.Version,
)
if err != nil {
return fmt.Errorf("update user: %w", err)
}
rows, _ := result.RowsAffected()
if rows == 0 {
return ErrConcurrentModification // another writer got there first
}
return nil
}
53. Time-Ordered IDs (ULID / KSUID) Link to heading
UUIDs are random and unsortable. Time-ordered IDs like ULIDs sort lexicographically by creation time, eliminating the need for secondary indexes on created_at and improving B-tree insert performance.
import "github.com/oklog/ulid/v2"
func NewID() string {
return ulid.Make().String()
// Example: "01HXYZ1234567890ABCDEFGHIJ"
// Sortable, unique, timestamp-embedded.
}
// IDs sort by creation time without an explicit timestamp column.
// SELECT * FROM events ORDER BY id -- gives chronological order
import ulid
def new_id() -> str:
return str(ulid.new())
# Lexicographically sortable, globally unique, embeds timestamp.
54. Soft-Delete and Tombstone Patterns Link to heading
Delete data logically rather than physically. A deleted_at timestamp marks rows as deleted while preserving them for audit trails, undo functionality, and referential integrity. Queries filter on deleted_at IS NULL by default.
func (r *repo) SoftDelete(ctx context.Context, id string) error {
_, err := r.db.ExecContext(ctx,
"UPDATE users SET deleted_at = NOW() WHERE id = $1 AND deleted_at IS NULL",
id,
)
return err
}
func (r *repo) GetByID(ctx context.Context, id string) (User, error) {
// Default queries exclude soft-deleted rows.
return r.queryOne(ctx,
"SELECT * FROM users WHERE id = $1 AND deleted_at IS NULL",
id,
)
}
Part IX: API, Transport, and CLI Link to heading
55. Middleware Chaining Link to heading
Middleware adds cross-cutting behavior to HTTP handlers — authentication, logging, CORS, rate limiting. Chain middleware by wrapping handlers so that each layer adds one concern.
type Middleware func(http.Handler) http.Handler
func Chain(handler http.Handler, middlewares ...Middleware) http.Handler {
// Apply in reverse so the first middleware in the list is outermost.
for i := len(middlewares) - 1; i >= 0; i-- {
handler = middlewares[i](handler)
}
return handler
}
// Usage: reads top-to-bottom as the request flow.
handler := Chain(
appHandler,
RecoveryMiddleware,
LoggingMiddleware(logger),
AuthMiddleware(verifier),
RateLimitMiddleware(limiter),
)
# FastAPI/Starlette: middleware applied in order.
app = FastAPI()
app.add_middleware(CORSMiddleware, allow_origins=["*"])
app.add_middleware(RequestLoggingMiddleware)
app.add_middleware(AuthMiddleware, verifier=token_verifier)
56. HTTP Transport Abstraction Layers Link to heading
Wrap external HTTP APIs behind an interface. The transport details — headers, authentication, URL construction — live in the client implementation, not in business logic.
type PaymentGateway interface {
Charge(ctx context.Context, customerID string, amount Money) (Receipt, error)
Refund(ctx context.Context, receiptID string) error
}
type stripeGateway struct {
client *http.Client
baseURL string
apiKey string
}
func (g *stripeGateway) Charge(ctx context.Context, customerID string, amount Money) (Receipt, error) {
body := buildChargeRequest(customerID, amount)
req, _ := http.NewRequestWithContext(ctx, http.MethodPost,
g.baseURL+"/charges", body)
req.Header.Set("Authorization", "Bearer "+g.apiKey)
req.Header.Set("Content-Type", "application/json")
resp, err := g.client.Do(req)
if err != nil {
return Receipt{}, fmt.Errorf("stripe charge: %w", err)
}
defer resp.Body.Close()
return parseReceipt(resp)
}
57. Adapter Pattern Around External Services Link to heading
Wrap third-party SDKs in an adapter that exposes only the operations your application needs. This isolates vendor lock-in to a single file and makes the dependency mockable for tests.
// Application interface — defined by your needs, not the vendor.
type ObjectStore interface {
Put(ctx context.Context, key string, data []byte) error
Get(ctx context.Context, key string) ([]byte, error)
Delete(ctx context.Context, key string) error
}
// Adapter wraps the AWS SDK.
type s3Store struct {
client *s3.Client
bucket string
}
func (s *s3Store) Put(ctx context.Context, key string, data []byte) error {
_, err := s.client.PutObject(ctx, &s3.PutObjectInput{
Bucket: &s.bucket,
Key: &key,
Body: bytes.NewReader(data),
})
return err
}
58. Strategy Pattern via Interfaces / Functions Link to heading
Select behavior at runtime by passing different implementations of an interface or different function values. This replaces switch statements with polymorphism.
// Strategy as interface.
type PricingStrategy interface {
CalculatePrice(base Money, quantity int) Money
}
type flatPricing struct{}
func (f flatPricing) CalculatePrice(base Money, qty int) Money {
return base.Multiply(qty)
}
type tieredPricing struct{ tiers []Tier }
func (t tieredPricing) CalculatePrice(base Money, qty int) Money {
// Apply volume discounts based on tiers.
for _, tier := range t.tiers {
if qty >= tier.MinQty {
return base.Multiply(qty).ApplyDiscount(tier.Discount)
}
}
return base.Multiply(qty)
}
// Strategy as function — simpler when the interface is a single method.
type PriceFunc func(base Money, qty int) Money
func WithBulkDiscount(threshold int, discount float64) PriceFunc {
return func(base Money, qty int) Money {
if qty >= threshold {
return base.Multiply(qty).ApplyDiscount(discount)
}
return base.Multiply(qty)
}
}
59. Command Pattern for CLI Applications Link to heading
Structure CLI commands as self-contained objects that encapsulate the operation, its arguments, and its execution. This separates argument parsing from business logic.
type Command struct {
Name string
Description string
Flags *flag.FlagSet
Run func(ctx context.Context, args []string) error
}
func migrateCommand(db *sql.DB) *Command {
fs := flag.NewFlagSet("migrate", flag.ExitOnError)
direction := fs.String("direction", "up", "migration direction")
return &Command{
Name: "migrate",
Description: "Run database migrations",
Flags: fs,
Run: func(ctx context.Context, args []string) error {
fs.Parse(args)
return runMigrations(ctx, db, *direction)
},
}
}
60. Cobra-Style CLI Composition Link to heading
For complex CLIs, compose commands into a tree. Each subcommand is self-contained with its own flags, help text, and execution function.
func main() {
root := &cobra.Command{Use: "app", Short: "Application management"}
root.AddCommand(
serveCmd(),
migrateCmd(),
seedCmd(),
workerCmd(),
)
if err := root.Execute(); err != nil {
os.Exit(1)
}
}
func serveCmd() *cobra.Command {
cmd := &cobra.Command{
Use: "serve",
Short: "Start the HTTP server",
RunE: func(cmd *cobra.Command, args []string) error {
port, _ := cmd.Flags().GetInt("port")
return startServer(cmd.Context(), port)
},
}
cmd.Flags().Int("port", 8080, "server port")
return cmd
}
61. Pipeline Stage Composition Link to heading
Model multi-step processing as a pipeline of composable stages. Each stage transforms input to output and can be tested independently.
type Stage[In, Out any] func(ctx context.Context, in In) (Out, error)
func Chain2[A, B, C any](first Stage[A, B], second Stage[B, C]) Stage[A, C] {
return func(ctx context.Context, in A) (C, error) {
var zero C
mid, err := first(ctx, in)
if err != nil {
return zero, err
}
return second(ctx, mid)
}
}
// Usage: compose stages into a pipeline.
pipeline := Chain2(
Chain2(validateStage, enrichStage),
publishStage,
)
result, err := pipeline(ctx, rawInput)
Part X: Events, Streams, and State Machines Link to heading
62. Informer / Watch / Event-Subscription Pattern Link to heading
Instead of polling, subscribe to change events from a data source. An informer caches the current state locally and receives incremental updates, reducing load on the source and enabling reactive processing.
type EventHandler func(eventType string, obj any)
type Informer struct {
client Client
cache sync.Map
handlers []EventHandler
}
func (inf *Informer) Run(ctx context.Context) error {
// Initial list — populate the cache.
items, err := inf.client.List(ctx)
if err != nil {
return err
}
for _, item := range items {
inf.cache.Store(item.ID, item)
}
// Watch for changes — receive incremental updates.
watcher, err := inf.client.Watch(ctx)
if err != nil {
return err
}
for {
select {
case event := <-watcher.Events():
inf.cache.Store(event.Object.ID, event.Object)
for _, h := range inf.handlers {
h(event.Type, event.Object)
}
case <-ctx.Done():
return nil
}
}
}
63. CQRS-Lite Read/Write Separation Link to heading
Separate read and write paths without a full CQRS infrastructure. Writes go through the domain model with validation. Reads use optimized queries that bypass the domain layer entirely.
// Write side: full domain validation.
type OrderCommandService struct {
repo OrderRepository
}
func (s *OrderCommandService) PlaceOrder(ctx context.Context, cmd PlaceOrderCmd) error {
order, err := NewOrder(cmd) // domain validation
if err != nil {
return err
}
return s.repo.Save(ctx, order)
}
// Read side: optimized query, no domain objects.
type OrderQueryService struct {
db *sql.DB
}
func (s *OrderQueryService) ListRecent(ctx context.Context, limit int) ([]OrderSummary, error) {
// Direct SQL — skips domain model overhead for read-only views.
rows, err := s.db.QueryContext(ctx,
`SELECT id, customer_name, total, status, created_at
FROM orders ORDER BY created_at DESC LIMIT $1`, limit)
if err != nil {
return nil, err
}
return scanOrderSummaries(rows)
}
64. Queue + Deduplication Processing Link to heading
Process messages from a queue with at-least-once delivery guarantees and deduplication to achieve exactly-once semantics at the application level.
func (w *Worker) processMessages(ctx context.Context) error {
for {
msg, err := w.queue.Receive(ctx)
if err != nil {
return err
}
// Deduplication: skip already-processed messages.
processed, err := w.dedup.HasProcessed(ctx, msg.ID)
if err != nil {
slog.Error("dedup check failed", "error", err)
msg.Nack()
continue
}
if processed {
msg.Ack() // already handled — skip silently
continue
}
if err := w.handle(ctx, msg); err != nil {
slog.Error("processing failed", "error", err, "msg_id", msg.ID)
msg.Nack()
continue
}
_ = w.dedup.MarkProcessed(ctx, msg.ID, 24*time.Hour)
msg.Ack()
}
}
65. Backpressure-Aware Processing Link to heading
When a producer is faster than a consumer, the system must apply backpressure — slowing the producer or buffering work — rather than dropping data or exhausting memory. Bounded channels and semaphores enforce backpressure naturally.
func (p *Pipeline) Run(ctx context.Context) error {
// Bounded buffer: producer blocks when buffer is full.
buffer := make(chan Event, 1000) // backpressure at 1000 pending events
g, ctx := errgroup.WithContext(ctx)
// Producer: blocks on buffer <- when full.
g.Go(func() error {
defer close(buffer)
return p.source.Stream(ctx, buffer)
})
// Consumer pool: drains buffer.
for range p.workers {
g.Go(func() error {
for event := range buffer {
if err := p.process(ctx, event); err != nil {
slog.Error("event failed", "error", err)
}
}
return nil
})
}
return g.Wait()
}
66. Streaming API Patterns Link to heading
For large result sets, stream data to the client rather than buffering the entire response in memory. In Go, this means writing to http.ResponseWriter incrementally. In Python, it means async generators.
func (h *Handler) StreamEvents(w http.ResponseWriter, r *http.Request) {
flusher, ok := w.(http.Flusher)
if !ok {
http.Error(w, "streaming not supported", http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "text/event-stream")
ctx := r.Context()
events := h.store.Subscribe(ctx)
for {
select {
case event := <-events:
fmt.Fprintf(w, "data: %s\n\n", event.JSON())
flusher.Flush()
case <-ctx.Done():
return
}
}
}
# Python: async generator for streaming responses.
async def stream_events(request: Request) -> StreamingResponse:
async def event_generator():
async for event in event_store.subscribe():
yield f"data: {event.json()}\n\n"
return StreamingResponse(event_generator(), media_type="text/event-stream")
67. Iterator Abstraction Patterns Link to heading
Abstract over paginated data sources with an iterator that fetches pages lazily. The consumer sees a simple Next() / __anext__() interface regardless of the underlying pagination mechanics.
type Iterator[T any] struct {
fetch func(cursor string) ([]T, string, error) // returns items, next cursor, error
buffer []T
cursor string
done bool
}
func (it *Iterator[T]) Next() (T, error) {
var zero T
if len(it.buffer) == 0 {
if it.done {
return zero, io.EOF
}
items, nextCursor, err := it.fetch(it.cursor)
if err != nil {
return zero, err
}
it.buffer = items
it.cursor = nextCursor
if nextCursor == "" {
it.done = true
}
}
item := it.buffer[0]
it.buffer = it.buffer[1:]
return item, nil
}
68. State Machine Modeling Link to heading
Model entities with complex lifecycle transitions as explicit state machines. Valid transitions are declared, and the machine rejects invalid ones. This eliminates scattered if status == "pending" checks throughout the codebase.
type State string
const (
StatePending State = "pending"
StateProcessing State = "processing"
StateCompleted State = "completed"
StateFailed State = "failed"
)
var validTransitions = map[State][]State{
StatePending: {StateProcessing, StateFailed},
StateProcessing: {StateCompleted, StateFailed},
// Completed and Failed are terminal — no outgoing transitions.
}
func (o *Order) TransitionTo(next State) error {
allowed, ok := validTransitions[o.Status]
if !ok {
return fmt.Errorf("no transitions from %s", o.Status)
}
for _, s := range allowed {
if s == next {
o.Status = next
return nil
}
}
return fmt.Errorf("invalid transition: %s → %s", o.Status, next)
}
Part XI: Caching, Pooling, and Performance Link to heading
69. Generic Typed Cache / Store Patterns Link to heading
Type-safe caching eliminates the interface{} casts that litter untyped cache implementations. With generics, the cache enforces types at compile time.
type Cache[K comparable, V any] struct {
mu sync.RWMutex
items map[K]cacheEntry[V]
ttl time.Duration
}
type cacheEntry[V any] struct {
value V
expiresAt time.Time
}
func (c *Cache[K, V]) Get(key K) (V, bool) {
c.mu.RLock()
defer c.mu.RUnlock()
entry, ok := c.items[key]
if !ok || time.Now().After(entry.expiresAt) {
var zero V
return zero, false
}
return entry.value, true
}
func (c *Cache[K, V]) Set(key K, value V) {
c.mu.Lock()
defer c.mu.Unlock()
c.items[key] = cacheEntry[V]{value: value, expiresAt: time.Now().Add(c.ttl)}
}
70. Object Pooling with sync.Pool Link to heading
Reuse expensive-to-allocate objects — byte buffers, JSON encoders, protobuf marshalers — instead of creating and garbage-collecting them per request. sync.Pool provides goroutine-safe object reuse.
var bufferPool = sync.Pool{
New: func() any {
return new(bytes.Buffer)
},
}
func Marshal(v any) ([]byte, error) {
buf := bufferPool.Get().(*bytes.Buffer)
defer func() {
buf.Reset()
bufferPool.Put(buf)
}()
if err := json.NewEncoder(buf).Encode(v); err != nil {
return nil, err
}
return bytes.Clone(buf.Bytes()), nil
}
71. Copy-on-Write Configuration / State Link to heading
Allow concurrent readers to access the current state without locking. When a writer updates state, it creates a new copy and atomically swaps the pointer. Readers always see a consistent snapshot.
type RouteTable struct {
routes atomic.Pointer[map[string]Handler]
}
func (rt *RouteTable) Lookup(path string) (Handler, bool) {
routes := *rt.routes.Load()
h, ok := routes[path]
return h, ok // lock-free read
}
func (rt *RouteTable) Update(path string, handler Handler) {
for {
old := rt.routes.Load()
newRoutes := make(map[string]Handler, len(*old)+1)
for k, v := range *old {
newRoutes[k] = v
}
newRoutes[path] = handler
if rt.routes.CompareAndSwap(old, &newRoutes) {
return
}
}
}
72. Read-Through / Write-Through Caching Link to heading
A read-through cache transparently loads missing entries from the source of truth. A write-through cache updates both the cache and the source in a single operation. The consumer does not manage cache coherence.
type ReadThroughCache[K comparable, V any] struct {
cache Cache[K, V]
loader func(ctx context.Context, key K) (V, error)
}
func (c *ReadThroughCache[K, V]) Get(ctx context.Context, key K) (V, error) {
if val, ok := c.cache.Get(key); ok {
return val, nil // cache hit
}
// Cache miss — load from source.
val, err := c.loader(ctx, key)
if err != nil {
var zero V
return zero, err
}
c.cache.Set(key, val)
return val, nil
}
73. Memory Reuse via Buffer Pools Link to heading
Allocate buffers from a pool sized to the common case. This reduces GC pressure on hot paths where allocation throughput matters.
// Pool of 4KB buffers — sized for typical HTTP request bodies.
var smallBufPool = sync.Pool{
New: func() any {
buf := make([]byte, 0, 4096)
return &buf
},
}
func ReadBody(r *http.Request) ([]byte, error) {
bufp := smallBufPool.Get().(*[]byte)
defer func() {
*bufp = (*bufp)[:0]
smallBufPool.Put(bufp)
}()
buf := *bufp
n, err := io.ReadFull(r.Body, buf[:cap(buf)])
if err != nil && err != io.ErrUnexpectedEOF {
return nil, err
}
return bytes.Clone(buf[:n]), nil
}
74. Immutable Event / Message Structs Link to heading
Events and messages should be immutable value types. Once created, they cannot be modified. This eliminates an entire class of concurrency bugs — if the struct cannot change, sharing it across goroutines is safe without synchronization.
// OrderPlaced is a value type — all fields are exported,
// but there are no setter methods and no pointer receivers.
type OrderPlaced struct {
OrderID string
CustomerID string
Total Money
Items []OrderItem // slice of values, not pointers
OccurredAt time.Time
}
// Constructor validates and freezes.
func NewOrderPlaced(order Order) OrderPlaced {
items := make([]OrderItem, len(order.Items))
copy(items, order.Items) // defensive copy — caller cannot mutate
return OrderPlaced{
OrderID: order.ID,
CustomerID: order.CustomerID,
Total: order.Total,
Items: items,
OccurredAt: time.Now(),
}
}
Part XII: Observability and Operations Link to heading
75. Structured Logging (Key/Value Logging) Link to heading
Log messages are for humans. Log fields are for machines. Structured logging emits key-value pairs that are searchable, filterable, and aggregatable — unlike free-text messages that require regex parsing.
// Go: slog (standard library since 1.21).
slog.Info("order processed",
"order_id", order.ID,
"customer_id", order.CustomerID,
"total", order.Total,
"duration_ms", time.Since(start).Milliseconds(),
"items_count", len(order.Items),
)
// Output: {"time":"...","level":"INFO","msg":"order processed","order_id":"abc","duration_ms":42}
# Python: structlog for structured key/value logging.
import structlog
logger = structlog.get_logger()
logger.info("order processed",
order_id=order.id,
customer_id=order.customer_id,
total=str(order.total),
duration_ms=elapsed_ms,
items_count=len(order.items),
)
76. Health Check and Readiness Probe Patterns Link to heading
Health checks report whether the service is alive. Readiness probes report whether the service can accept traffic. A service that is alive but not ready (still warming caches, running migrations) should receive no traffic from the load balancer.
func (s *Server) healthHandler(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK) // alive
}
func (s *Server) readyHandler(w http.ResponseWriter, r *http.Request) {
checks := map[string]error{
"database": s.db.PingContext(r.Context()),
"cache": s.cache.Ping(r.Context()),
}
for name, err := range checks {
if err != nil {
slog.Error("readiness check failed", "check", name, "error", err)
http.Error(w, fmt.Sprintf("%s: unhealthy", name), http.StatusServiceUnavailable)
return
}
}
w.WriteHeader(http.StatusOK)
}
77. Metrics Instrumentation at Boundaries Link to heading
Instrument at service boundaries — where requests enter, where external calls leave, where messages are consumed. This captures the data that matters for SLOs: request rate, error rate, and latency distributions.
func MetricsMiddleware(next http.Handler) http.Handler {
requestDuration := prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: "http_request_duration_seconds",
Buckets: prometheus.DefBuckets,
},
[]string{"method", "path", "status"},
)
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
start := time.Now()
wrapped := &statusRecorder{ResponseWriter: w, status: 200}
next.ServeHTTP(wrapped, r)
requestDuration.WithLabelValues(
r.Method,
r.URL.Path,
strconv.Itoa(wrapped.status),
).Observe(time.Since(start).Seconds())
})
}
78. OpenTelemetry Tracing Propagation Link to heading
Distributed tracing requires propagating trace context across service boundaries. Each service extracts the trace ID from incoming requests and injects it into outgoing calls. This produces end-to-end trace views across the entire request path.
func TracingMiddleware(next http.Handler) http.Handler {
tracer := otel.Tracer("api")
propagator := otel.GetTextMapPropagator()
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Extract trace context from incoming request headers.
ctx := propagator.Extract(r.Context(), propagation.HeaderCarrier(r.Header))
ctx, span := tracer.Start(ctx, r.URL.Path)
defer span.End()
span.SetAttributes(
attribute.String("http.method", r.Method),
attribute.String("http.url", r.URL.String()),
)
next.ServeHTTP(w, r.WithContext(ctx))
})
}
79. Observable, Operable Service Design Link to heading
Observability is not an add-on — it is a first-class design constraint. Every service should expose health endpoints, emit structured logs, publish metrics, propagate traces, and surface configuration at runtime. An operable service answers “what is happening right now?” without requiring SSH access or log scraping.
func NewOperableServer(cfg Config, app http.Handler) *http.ServeMux {
mux := http.NewServeMux()
// Application routes.
mux.Handle("/", app)
// Operations routes — always available, not behind auth.
mux.HandleFunc("/healthz", healthHandler)
mux.HandleFunc("/readyz", readyHandler)
mux.Handle("/metrics", promhttp.Handler())
mux.HandleFunc("/debug/pprof/", pprof.Index)
mux.HandleFunc("/config", configHandler(cfg)) // expose non-secret config
return mux
}
Part XIII: Testing Link to heading
80. Table-Driven Tests Link to heading
Define test cases as data in a slice. Each entry specifies inputs, expected outputs, and a descriptive name. The test loop iterates over cases, making it trivial to add new scenarios without duplicating test structure.
func TestParseAmount(t *testing.T) {
tests := []struct {
name string
input string
want int64
wantErr bool
}{
{name: "whole dollars", input: "42", want: 4200},
{name: "cents", input: "42.50", want: 4250},
{name: "zero", input: "0", want: 0},
{name: "negative", input: "-10", want: -1000},
{name: "invalid", input: "abc", wantErr: true},
{name: "empty", input: "", wantErr: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := ParseAmount(tt.input)
if tt.wantErr {
require.Error(t, err)
return
}
require.NoError(t, err)
assert.Equal(t, tt.want, got)
})
}
}
# Python: pytest.mark.parametrize achieves the same pattern.
import pytest
@pytest.mark.parametrize("input_str, expected", [
("42", 4200),
("42.50", 4250),
("0", 0),
("-10", -1000),
])
def test_parse_amount(input_str: str, expected: int) -> None:
assert parse_amount(input_str) == expected
@pytest.mark.parametrize("input_str", ["abc", ""])
def test_parse_amount_invalid(input_str: str) -> None:
with pytest.raises(ValueError):
parse_amount(input_str)
81. Subtests with t.Run() Link to heading
Group related assertions under named subtests. Each subtest runs independently — a failure in one does not skip the others — and the name appears in test output for precise failure identification.
func TestUserService(t *testing.T) {
svc := setupTestService(t)
t.Run("create", func(t *testing.T) {
user, err := svc.Create(ctx, validUser)
require.NoError(t, err)
assert.NotEmpty(t, user.ID)
})
t.Run("create_duplicate_email", func(t *testing.T) {
_, err := svc.Create(ctx, validUser) // same email
require.Error(t, err)
assert.True(t, errors.Is(err, ErrConflict))
})
t.Run("get_nonexistent", func(t *testing.T) {
_, err := svc.GetByID(ctx, "nonexistent")
require.Error(t, err)
assert.True(t, errors.Is(err, ErrNotFound))
})
}
82. Golden-File Testing Link to heading
Compare output against a known-good reference file. When the output changes intentionally, update the golden file with a flag. This pattern excels for testing serialization, code generation, and template rendering.
var update = flag.Bool("update", false, "update golden files")
func TestRenderTemplate(t *testing.T) {
got := renderInvoice(testOrder)
goldenPath := filepath.Join("testdata", t.Name()+".golden")
if *update {
os.WriteFile(goldenPath, []byte(got), 0644)
return
}
want, err := os.ReadFile(goldenPath)
require.NoError(t, err)
assert.Equal(t, string(want), got)
}
// Run with: go test -update to regenerate golden files.
83. Fake / Mock Implementations Over Mocking Frameworks Link to heading
Write explicit fake implementations rather than generating mocks with frameworks. Fakes are readable, debuggable, and behave like the real thing — they maintain state, enforce invariants, and can be shared across tests.
// Fake — a complete in-memory implementation.
type FakeUserRepo struct {
mu sync.Mutex
users map[string]User
}
func NewFakeUserRepo() *FakeUserRepo {
return &FakeUserRepo{users: make(map[string]User)}
}
func (f *FakeUserRepo) GetByID(_ context.Context, id string) (User, error) {
f.mu.Lock()
defer f.mu.Unlock()
user, ok := f.users[id]
if !ok {
return User{}, ErrNotFound
}
return user, nil
}
func (f *FakeUserRepo) Save(_ context.Context, user User) error {
f.mu.Lock()
defer f.mu.Unlock()
f.users[user.ID] = user
return nil
}
// Usage in tests — no mocking framework, no generated code.
func TestUserService_GetUser(t *testing.T) {
repo := NewFakeUserRepo()
repo.Save(context.Background(), User{ID: "1", Email: "test@example.com"})
svc := NewUserService(repo, noopLogger{})
user, err := svc.GetByID(context.Background(), "1")
require.NoError(t, err)
assert.Equal(t, "test@example.com", user.Email)
}
# Python: same principle — hand-written fakes, not mock.patch.
class FakeUserRepo:
def __init__(self) -> None:
self._users: dict[str, User] = {}
async def get_by_id(self, user_id: str) -> User | None:
return self._users.get(user_id)
async def save(self, user: User) -> None:
self._users[user.id] = user
async def test_get_user() -> None:
repo = FakeUserRepo()
await repo.save(User(id="1", email="test@example.com"))
svc = UserService(repo=repo)
user = await svc.get_by_id("1")
assert user.email == "test@example.com"
84. Test Helper Packages and Fixtures Link to heading
Extract shared test setup — database connections, fixture loading, assertion helpers — into dedicated test helper packages. Helpers reduce duplication and make individual test files focused on behavior, not infrastructure.
// testutil/db.go — shared test infrastructure.
package testutil
func NewTestDB(t *testing.T) *sql.DB {
t.Helper()
db, err := sql.Open("postgres", os.Getenv("TEST_DATABASE_URL"))
require.NoError(t, err)
t.Cleanup(func() { db.Close() })
// Run migrations.
require.NoError(t, migrate.Up(db))
// Clean state between tests.
t.Cleanup(func() { truncateAll(t, db) })
return db
}
func LoadFixture(t *testing.T, name string) []byte {
t.Helper()
data, err := os.ReadFile(filepath.Join("testdata", name))
require.NoError(t, err)
return data
}
Part XIV: Architecture and Package Design Link to heading
85. Single-Responsibility Packages Link to heading
A package should do one thing. A package named utils or helpers is a code smell — it is a junk drawer that grows without bound. Name packages after what they provide: auth, invoice, postgres, retry.
// Bad: everything in one package.
pkg/
├── utils/
│ ├── string_helpers.go
│ ├── http_client.go
│ ├── retry.go
│ └── database.go
// Good: each package has a single responsibility.
pkg/
├── retry/ // retry logic
│ └── retry.go
├── httputil/ // HTTP client helpers
│ └── client.go
├── postgres/ // PostgreSQL client
│ └── client.go
└── auth/ // authentication
├── token.go
└── middleware.go
86. Dependency Boundaries via internal Packages Link to heading
Go’s internal/ directory restricts imports to the parent module. Use it to mark packages as private to a service — other services in the monorepo cannot depend on internal implementation details.
apps/
├── orders-api/
│ ├── cmd/main.go
│ └── internal/ # only orders-api can import these
│ ├── handlers/
│ ├── services/
│ └── repos/
├── payments-api/
│ ├── cmd/main.go
│ └── internal/ # only payments-api can import these
│ ├── handlers/
│ └── services/
pkg/ # shared code — importable by any service
├── auth/
├── retry/
└── logging/
87. Package-Oriented Architecture Over Layered Monoliths Link to heading
Organize by domain, not by technical layer. A user/ package that contains the handler, service, repository, and models for user operations is more cohesive than scattering user logic across handlers/, services/, and models/ directories.
// Layer-oriented (vertical slicing required to understand any feature):
handlers/
├── user_handler.go
├── order_handler.go
services/
├── user_service.go
├── order_service.go
models/
├── user.go
├── order.go
// Package-oriented (each feature is self-contained):
user/
├── handler.go
├── service.go
├── repository.go
├── model.go
├── user_test.go
order/
├── handler.go
├── service.go
├── repository.go
├── model.go
├── order_test.go
88. Hexagonal / Ports-and-Adapters-Lite Architecture Link to heading
The domain layer defines ports (interfaces). Adapters implement those interfaces for specific technologies. The domain never imports adapter packages — dependency flows inward.
// Port: defined in the domain package.
package order
type Repository interface {
Save(ctx context.Context, order Order) error
GetByID(ctx context.Context, id string) (Order, error)
}
type NotificationSender interface {
SendOrderConfirmation(ctx context.Context, order Order) error
}
// Adapter: implements the port with PostgreSQL.
package postgres
type OrderRepository struct{ db *sql.DB }
func (r *OrderRepository) Save(ctx context.Context, o order.Order) error {
// PostgreSQL-specific implementation
}
// Adapter: implements the port with SendGrid.
package sendgrid
type Notifier struct{ client *sendgrid.Client }
func (n *Notifier) SendOrderConfirmation(ctx context.Context, o order.Order) error {
// SendGrid-specific implementation
}
89. Domain-Specific Package Naming Link to heading
Name packages after the domain concept they represent, not after the technology they use. A package named invoice is stable across technology changes. A package named mongoInvoice couples the name to an implementation detail.
// Bad: technology in the package name.
mongorepo/
├── user.go
├── order.go
rediscache/
├── session.go
// Good: domain names, technology is an implementation detail.
user/
├── repository.go // interface
├── postgres.go // PostgreSQL implementation
├── mongo.go // MongoDB implementation (if needed)
order/
├── repository.go
├── postgres.go
90. Thin Orchestration / Fat Domain Logic Link to heading
Controllers and workflows should be thin orchestration layers that delegate to domain services. The domain contains the business rules, validation, and state transitions. If a controller has an if statement that checks business state, the logic is in the wrong place.
// Thin controller — no business logic.
func (h *Handler) PlaceOrder(w http.ResponseWriter, r *http.Request) {
var req PlaceOrderRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, "invalid request", http.StatusBadRequest)
return
}
order, err := h.service.PlaceOrder(r.Context(), req.toDomain())
if err != nil {
writeError(w, err)
return
}
writeJSON(w, http.StatusCreated, toOrderResponse(order))
}
// Fat domain — all business rules live here.
func (s *Service) PlaceOrder(ctx context.Context, order Order) (Order, error) {
if err := order.Validate(); err != nil {
return Order{}, err
}
if err := s.inventory.Reserve(ctx, order.Items); err != nil {
return Order{}, fmt.Errorf("reserve inventory: %w", err)
}
order.Status = StatusConfirmed
order.ConfirmedAt = time.Now()
return s.repo.Save(ctx, order)
}
91. Boundary-Focused Validation Link to heading
Validate at system boundaries — where data enters from external sources. Once data passes the boundary and becomes a domain entity, trust it. Internal function calls should not re-validate what the boundary already verified.
// Validate at the API boundary.
func (h *Handler) CreateUser(w http.ResponseWriter, r *http.Request) {
var req CreateUserRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, "invalid JSON", http.StatusBadRequest)
return
}
// Boundary validation — exhaustive.
if err := req.Validate(); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
// Domain layer trusts the input — no re-validation.
user, err := h.service.CreateUser(r.Context(), req.toDomain())
// ...
}
func (r CreateUserRequest) Validate() error {
var errs []error
if r.Email == "" {
errs = append(errs, errors.New("email is required"))
}
if len(r.Name) < 2 {
errs = append(errs, errors.New("name must be at least 2 characters"))
}
return errors.Join(errs...)
}
Part XV: Generics and Modern Go Link to heading
92. Generics-Based Reusable Utilities Link to heading
Go 1.18+ generics eliminate the need for per-type utility functions. A single Map, Filter, or Contains function works across all types.
func Map[T, U any](items []T, fn func(T) U) []U {
result := make([]U, len(items))
for i, item := range items {
result[i] = fn(item)
}
return result
}
func Filter[T any](items []T, predicate func(T) bool) []T {
var result []T
for _, item := range items {
if predicate(item) {
result = append(result, item)
}
}
return result
}
func Contains[T comparable](items []T, target T) bool {
for _, item := range items {
if item == target {
return true
}
}
return false
}
// Usage: type-safe, no casting.
names := Map(users, func(u User) string { return u.Name })
active := Filter(users, func(u User) bool { return u.Active })
93. Functional Slice/Map/Filter Helpers with Generics Link to heading
Extend basic generics into a collection of slice and map operations that compose naturally. These replace common loop patterns with declarative expressions.
func GroupBy[T any, K comparable](items []T, key func(T) K) map[K][]T {
groups := make(map[K][]T)
for _, item := range items {
k := key(item)
groups[k] = append(groups[k], item)
}
return groups
}
func Reduce[T, U any](items []T, initial U, fn func(U, T) U) U {
acc := initial
for _, item := range items {
acc = fn(acc, item)
}
return acc
}
func Keys[K comparable, V any](m map[K]V) []K {
keys := make([]K, 0, len(m))
for k := range m {
keys = append(keys, k)
}
return keys
}
// Usage: group orders by status, sum totals.
byStatus := GroupBy(orders, func(o Order) string { return string(o.Status) })
total := Reduce(orders, Money(0), func(sum Money, o Order) Money { return sum + o.Total })
Part XVI: Remaining Patterns Link to heading
94. Controller-Runtime Reconcile Contracts Link to heading
In Kubernetes controller development, the reconcile function is the central contract. It takes the desired state (from the custom resource) and converges actual state toward it. The runtime handles requeueing, error retry, and leader election.
type Reconciler struct {
client client.Client
}
func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
var resource MyResource
if err := r.client.Get(ctx, req.NamespacedName, &resource); err != nil {
return ctrl.Result{}, client.IgnoreNotFound(err)
}
// Converge actual → desired.
if resource.Status.Phase != resource.Spec.DesiredPhase {
if err := r.converge(ctx, &resource); err != nil {
return ctrl.Result{RequeueAfter: 30 * time.Second}, err
}
}
return ctrl.Result{}, nil
}
95. Shared Informer Cache Patterns Link to heading
In Kubernetes operators, shared informers watch API resources and maintain a local cache. Multiple controllers share the same informer to avoid duplicate API server load.
func setupInformer(ctx context.Context, client kubernetes.Interface) cache.SharedIndexInformer {
factory := informers.NewSharedInformerFactory(client, 30*time.Second)
podInformer := factory.Core().V1().Pods().Informer()
podInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: func(obj interface{}) { handlePodAdd(obj.(*v1.Pod)) },
UpdateFunc: func(old, new interface{}) { handlePodUpdate(old.(*v1.Pod), new.(*v1.Pod)) },
DeleteFunc: func(obj interface{}) { handlePodDelete(obj.(*v1.Pod)) },
})
factory.Start(ctx.Done())
factory.WaitForCacheSync(ctx.Done())
return podInformer
}
96. Eventual Consistency Reconciliation Link to heading
In distributed systems, services will disagree temporarily. A reconciliation process periodically compares states across services and resolves discrepancies, converging the system toward consistency.
func (r *Reconciler) reconcileInventory(ctx context.Context) error {
// Compare order service state with inventory service state.
orders, _ := r.orderStore.ListReserved(ctx)
inventory, _ := r.inventoryStore.ListAllocations(ctx)
orderSet := toSet(orders, func(o Order) string { return o.ID })
allocSet := toSet(inventory, func(a Allocation) string { return a.OrderID })
// Orders with reservations but no allocation — create allocation.
for id := range orderSet {
if _, ok := allocSet[id]; !ok {
slog.Warn("missing allocation, creating", "order_id", id)
_ = r.inventoryStore.Allocate(ctx, id)
}
}
// Allocations with no matching order — release.
for id := range allocSet {
if _, ok := orderSet[id]; !ok {
slog.Warn("orphaned allocation, releasing", "order_id", id)
_ = r.inventoryStore.Release(ctx, id)
}
}
return nil
}
97. Event-Driven Processing Pipelines Link to heading
Process events through a series of handlers, where each handler can transform, filter, enrich, or route events. The pipeline decouples event production from consumption.
type EventHandler func(ctx context.Context, event Event) (Event, error)
type EventPipeline struct {
handlers []EventHandler
}
func (p *EventPipeline) Process(ctx context.Context, event Event) error {
current := event
for _, handler := range p.handlers {
next, err := handler(ctx, current)
if err != nil {
return fmt.Errorf("pipeline stage: %w", err)
}
current = next
}
return nil
}
// Compose pipelines from focused handlers.
pipeline := &EventPipeline{
handlers: []EventHandler{
validateEvent,
enrichWithMetadata,
filterDuplicates,
routeToDestination,
},
}
98. Stream Processing Worker Patterns Link to heading
Long-running workers that consume from Kafka, Redis Streams, or SQS need consistent lifecycle management: connect, consume, process, acknowledge, and shut down cleanly.
type StreamWorker struct {
consumer Consumer
handler func(ctx context.Context, msg Message) error
}
func (w *StreamWorker) Run(ctx context.Context) error {
for {
msg, err := w.consumer.Receive(ctx)
if err != nil {
if ctx.Err() != nil {
return nil // clean shutdown
}
slog.Error("receive failed", "error", err)
continue
}
if err := w.handler(ctx, msg); err != nil {
slog.Error("handle failed", "error", err, "msg_id", msg.ID)
msg.Nack()
continue
}
msg.Ack()
}
}
99. Zero-Allocation Hot-Path Optimization Link to heading
On performance-critical paths, avoid allocations by reusing buffers, using stack-allocated arrays, and avoiding interface conversions that cause heap escapes. Profile first — optimize only what the profiler identifies as hot.
// Stack-allocated array for small, known-size operations.
func formatKey(prefix, id string) string {
// Avoid fmt.Sprintf — it allocates.
var buf [128]byte // stack-allocated
n := copy(buf[:], prefix)
n += copy(buf[n:], ":")
n += copy(buf[n:], id)
return string(buf[:n])
}
// Pre-allocated slice for known-capacity operations.
func collectIDs(users []User) []string {
ids := make([]string, 0, len(users)) // pre-allocate capacity
for _, u := range users {
ids = append(ids, u.ID)
}
return ids
}
100. Consistent Hashing and Work Distribution Link to heading
For distributed work queues and caches, consistent hashing assigns keys to workers with minimal redistribution when workers join or leave. Each worker is responsible for a deterministic subset of the keyspace.
type ConsistentHash struct {
ring []uint32
nodes map[uint32]string
replicas int
}
func (ch *ConsistentHash) Add(node string) {
for i := range ch.replicas {
hash := hashKey(fmt.Sprintf("%s-%d", node, i))
ch.ring = append(ch.ring, hash)
ch.nodes[hash] = node
}
sort.Slice(ch.ring, func(i, j int) bool { return ch.ring[i] < ch.ring[j] })
}
func (ch *ConsistentHash) Get(key string) string {
hash := hashKey(key)
idx := sort.Search(len(ch.ring), func(i int) bool {
return ch.ring[i] >= hash
})
if idx == len(ch.ring) {
idx = 0
}
return ch.nodes[ch.ring[idx]]
}
Summary Link to heading
Patterns are not rules. They are tools — each appropriate in specific contexts and counterproductive in others. A worker pool is overhead for a CLI that processes ten items. A circuit breaker is unnecessary for a service that calls only itself. The judgment lies in knowing when a pattern earns its complexity.
What these hundred patterns share is a common orientation: make the implicit explicit. Explicit dependency injection over hidden globals. Explicit error classification over string matching. Explicit lifecycle management over fire-and-forget goroutines. Explicit interface contracts over duck typing and hope.
The difference between a codebase that scales and one that collapses under its own weight is not the cleverness of any individual solution. It is the consistency of the patterns that hold the system together. When every service handles errors the same way, shuts down the same way, and exposes health checks the same way, the system becomes navigable. A new engineer can read one service and understand all of them.
Build the vocabulary. Apply it consistently. Let the patterns compound.
