It looked innocent. A transaction ID embedded in a metric name. A session token baked into a log field. A request hash used as an analytics event name. The system accepted it, stored it, and indexed it without question.

Nobody ever queried it. Nobody ever would. But it kept arriving. Millions of them. Each one adding weight to a system that was never designed to carry it.

This is the story of random string pollution. And one way to stop it before it gets in.

The Problem Repeats Itself

Random strings cause the same class of damage across different parts of your stack. The pattern is always the same. A developer instruments something with a unique identifier baked into a key or a name. The system treats every unique value as a new entry. Data explodes. Nothing useful gets added.

Here is where you have probably seen this play out.

Metrics pipelines. Your service emits api.response.txn_a3f9b2c1.duration for every transaction. One million transactions a day means one million new time series today alone. Your monitoring system slows down. Storage balloons. Nobody ever queries a dashboard by a specific transaction hash.

Log indexing. Your application ships structured logs to Elasticsearch. A developer adds a field called request_7f3a9b2c to capture per request context. Elasticsearch creates a new field mapping for every unique key it sees. Your index mapping explodes into thousands of dynamic fields. Queries slow to a crawl. Storage costs spike. The fields never appear in any search.

Distributed tracing. Your tracing system records span names like process.job_4e8d1a3f.execute for background jobs. Every job run produces a new span name. Your trace search index fills up with spans that share no common name to group or filter by. Aggregations become meaningless. Finding slow jobs requires scrolling through thousands of one off entries.

Analytics event tracking. Your frontend fires events named checkout.session_9c2f7b1a.started for every user session. Your analytics platform stores each event name as a distinct category. Reports fragment into noise. No analyst can aggregate checkout behavior across sessions because every session has a different event name.

The root cause is the same in every case. A random string got embedded in something that should only contain human readable words. The system did not know the difference.

The First Instinct

You write rules. You know what random identifiers look like. UUIDs, hex strings, base58 encoded tokens. You write a regex for each and drop it at the ingestion boundary.

[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}

[0-9a-f]{12,}

For a while this works. The obvious offenders get caught. Things calm down.

Then someone switches to a shorter ID format. Or they move to base62 encoding. Or they start using numeric sequences that look nothing like hex. A new pattern slips through every few weeks. Another rule gets added. The blocklist grows. Someone eventually adds a comment that says "do not touch this, nobody knows why it is here."

You are patching holes in a leaking pipe. What you need is a different pipe.

A Different Question

Instead of asking "does this match a known bad pattern," ask a better question.

Does this string look like something a human would write?

Human readable strings like api.response.duration, checkout.started, or process.job.execute come from natural language. Letters in natural language are not random. They follow patterns. In English, th, er, and in are extremely common consecutive letter pairs. xq, zk, and vj almost never appear in real words.

Random strings like UUIDs and hashes ignore these patterns. Their characters distribute more evenly and more chaotically. You can measure that difference. And once you can measure it, you can act on it.

How It Works

A bigram is a pair of consecutive characters. In the word metric, the bigrams are me, et, tr, ri, and ic. There are 676 possible bigrams across the lowercase alphabet.

Build a probability map from real English. Take an English dictionary or a large text corpus. For every word, extract all its bigrams. Count how often each of the 676 possible pairs appears across all words. Normalize the counts to sit between 0 and 1. What you get is a map that tells you how likely any given letter pair is to appear in real English text.

"th" -> 0.92
"er" -> 0.87
"xq" -> 0.01
"zk" -> 0.00

You compute this map once, write it to a file, and load it into memory at startup. It is 676 entries. A few kilobytes. It never changes.

Score incoming strings. When a string arrives, tokenize it by splitting on ., _, and other delimiters to get individual segments. For each segment, extract its bigrams. Look up each bigram in the map. Apply a threshold. Any bigram below the threshold counts as random looking. Count random bigrams against normal looking ones across all tokens.

Input: api.response.txn_a3f9b2c1.duration

Tokens: api, response, txn, a3f9b2c1, duration

Bigrams from "a3f9b2c1" (letters only: a, f, b, c):
  "af" -> 0.04   random
  "fb" -> 0.01   random
  "bc" -> 0.02   random

Bigrams from "response":
  "re" -> 0.81   normal
  "es" -> 0.74   normal
  "sp" -> 0.55   normal

Verdict: random bigrams exceed threshold, reject

Act on the result. You reject the input, emit a detection event, and surface which services or clients are the biggest contributors. You make the threshold configurable so you can tune sensitivity without touching rule definitions.

Why This Works Across Your Stack

You can apply this check anywhere in your system that accepts a string key or name from an external source.

Drop it at your metrics ingestion layer and stop cardinality explosion before it touches your time series database. Drop it at your log pipeline and prevent dynamic field mapping from blowing up your Elasticsearch index. Drop it at your tracing collector and keep your span names human readable and groupable. Drop it at your analytics ingest endpoint and keep your event taxonomy clean enough to actually report on.

The logic is the same in every case. The map does not change. Only the strings being checked are different.

A regex blocklist says "I know exactly what bad looks like." The bigram approach says "I know what readable looks like, and I reject what does not match." That position holds up as your system grows and as the patterns of abuse change over time.

The configurable threshold matters too. Not every string in your system is a clean English word. Abbreviations, domain specific terms, and short tokens sometimes score lower than you expect. You tune the threshold once per use case and move on.

The Gate Holds

Back to that string. api.response.txn_a3f9b2c1d2e3.duration.

It arrives at your ingestion layer. Your system tokenizes the name, extracts bigrams, and runs the probabilities. The random looking pairs dominate. A detection event fires. The string goes nowhere.

Its siblings follow. Thousands of them across your metrics pipeline, your log shipper, your tracing collector, your analytics endpoint. Each one scoring poorly. None of them getting through. Your storage stays predictable. Your indexes stay clean. Your dashboards load.

The services sending them have no idea. That is fine. Someone will notice the detection events and investigate. But the damage stopped the moment it would have started.

What was the motivation? Most engineers have felt this pain at least once. You are building a service that needs a strongly consistent, distributed configuration store. You reach for Redis, Etcd, or ZooKeeper — and then spend the next two weeks onboarding a new dependency, wiring up a separate deployment, managing its credentials, and debugging. The data requirements are modest - a few dozen keys, each a small string. The operational weight is disproportionate.

What if the store was just part of your service? Not a sidecar, not a separate container — but a library you integrate into your application. That is exactly the problem AtomDB was built to solve.

AtomDB is a distributed key-value store built on top of Apache Ratis, the Java implementation of the Raft consensus protocol. It ships as a Dropwizard bundle or a Standard Java Client, which can start a fully replicated Raft group within the same JVM process. There is no separate binary to run, no extra Dockerfile, and no extra deploy step. The cluster forms itself, heals itself, and can even grow and shrink automatically while your service is live.

This post is the engineering narrative behind that system: how it works, why it is designed the way it is, and what the code looks like from the inside.

Embedded Model

Traditional key-value stores like Redis or Etcd are designed as an external cluster which you connect to over a network. Your service is a client of that external system. That model carries a hidden tax - every read or write crosses a network boundary, the store must be independently deployed and monitored, and your service's availability now depends on the health of an entirely separate system.

AtomDB flips the model. Instead of your service being a client of an external store, the store is embedded inside your service. Each instance of your application carries its own Raft node. Together those replicas form a Raft cluster and maintain a shared, consistent state without ever leaving the JVM process.

The practical consequence: every instance of your service participates in the cluster. When you scale your service from 3 pods to 5, two new Raft peers come online automatically. When you deploy your service, you deploy the store too.

Registering AtomDB bundle in your application looks exactly like adding any other Dropwizard bundle.

bootstrap.addBundle(new AtomDBBundle<YourConfiguration>() {

    @Override
    public AtomDBBundleConfig getAtomDBBundleConfig(YourConfiguration config) {
        return config.getAtomDBBundleConfig();
    }

    @Override
    public int getApplicationPort(YourConfiguration config) {
        DefaultServerFactory serverFactory = (DefaultServerFactory) config.getServerFactory();
        for (ConnectorFactory connector : serverFactory.getApplicationConnectors()) {
            if (connector instanceof HttpConnectorFactory httpConnector) {
                return httpConnector.getPort();
            }
        }
        return -1;
    }
});

That single addBundle call wires together the Raft group, the state machine, the HTTP operators, and a Feign-based client — all from the YAML configuration your service already reads at startup.

In case you are not using Dropwizard, you can use the provided atomdb-client to integrate it in your Java application.

Raft Consensus: Why Every Write Needs Majority Agreement

Before understanding how AtomDB stores a value, you need to understand Raft's core guarantee. Raft is a consensus algorithm that ensures a cluster of nodes agrees on a single, ordered log of operations. In a cluster of N nodes, every write must be acknowledged by a strict majority (N/2 + 1 nodes) before it is considered committed. This majority is called the quorum.

For a 3-node AtomDB cluster, that means at least 2 nodes must confirm every PUT before the client sees a success response. For 5 nodes, at least 3 must confirm. Note that if you have a cluster of 100 Dropwizard applications, you don't have to run a 100 node AtomDB cluster. You can form a smaller 3 node cluster where your metadata can reside.

Source: https://www.mydistributed.systems/2021/04/raft.html

Reads Route to the Leader

In AtomDB, reads (GET) are always routed to the current Raft leader. This is intentional and deliberate. Because the leader is the node that manages log replication, it always has the most up-to-date committed state. By routing reads through the leader, AtomDB guarantees read-after-write consistency: if your client successfully commits a PUT, any subsequent GET on any node will observe that value.

This is markedly different from eventually-consistent systems where a write to Node 1 might not be immediately visible on a GET to Node 2. In AtomDB, consistency is a hard guarantee, not a probabilistic one.

The Ratis query() path is non-replicated — it answers directly from the leader's in-memory state without creating a log entry, keeping reads fast while preserving the consistency guarantee.

// In KeyValueStateMachine.java
@Override
public CompletableFuture<Message> query(Message request) {
    final String logData = request.getContent().toStringUtf8();
    final String[] parts = logData.split(":", 3);

    if ("GET".equals(parts[0]) && parts.length >= 2) {
        final String key = parts[1];
        final String value = keyValueStore.getOrDefault(key, "NOT_FOUND");
        return CompletableFuture.completedFuture(Message.valueOf(value));
    }
    return CompletableFuture.completedFuture(Message.valueOf("INVALID_QUERY"));
}

The State Machine: Writes Are Applied Only After Quorum

The Raft log is not the data store — it is the ordered record of operations that transform the data store. The actual key-value pairs live in the KeyValueStateMachine, a class that extends Ratis's BaseStateMachine. Crucially, applyTransaction is called by the Raft runtime only after the log entry has been replicated to a majority of nodes and committed. Your in-memory ConcurrentHashMap is never touched by a write that hasn't cleared quorum.

// In KeyValueStateMachine.java
private final ConcurrentHashMap<String, String> keyValueStore = new ConcurrentHashMap<>();

@Override
public CompletableFuture<Message> applyTransaction(TransactionContext trx) {
    final RaftProtos.LogEntryProto entry = trx.getLogEntry();
    final String logData = entry.getStateMachineLogEntry()
            .getLogData().toStringUtf8();

    // Parse command: PUT:key:value
    final String[] parts = logData.split(":", 3);
    if ("PUT".equals(parts[0]) && parts.length == 3) {
        keyValueStore.put(parts[1], parts[2]);
        reply = Message.valueOf("OK");
    }

    // This is the contract with Ratis: record the highest applied index
    updateLastAppliedTermIndex(entry.getTerm(), entry.getIndex());
    return CompletableFuture.completedFuture(reply);
}

The call to updateLastAppliedTermIndex at the end is not optional. It tells Raft which log entry this node has processed, so that in the event of a crash the runtime knows where to resume replay. Forget that call and you break the replay invariant — the node will re-apply entries it already applied, potentially corrupting state.

Think of it like Kafka KRaft, which replaced ZooKeeper with a Raft-based metadata quorum. Every broker metadata change (partition assignment, ISR list update, config change) is a log entry. The brokers' in-memory metadata is applied only after entries commit. applyTransaction in AtomDB is the direct counterpart of KRaft's metadata record application — and takeSnapshot maps cleanly onto KRaft's metadata snapshot mechanism.

The Control Plane: Dynamic Peer Management Without Manual Intervention

Raft itself is a consensus protocol, not an operations framework. It will faithfully replicate log entries once you tell it who the peers are — but it has no notion of go discover three more nodes on these ports and add them to the cluster. That operational intelligence is AtomDB's control plane.

Bootstrap: Who Goes First?

When multiple nodes start simultaneously, exactly one of them must bootstrap the Raft group; the others must join it. Choosing the wrong bootstrapper (or having two nodes both bootstrap) corrupts the group. AtomDB resolves this with a deterministic election inside ClusterManager:

1. clusterService.awaitDiscovery() blocks until at least memberSize peers are reachable on their admin ports (via TCP probe).

2. From the list of reachable peers, sorted deterministically, the first node in the list becomes the candidate bootstrapper.

3. Before bootstrapping, the candidate probes all other peers' HTTP /cluster/v1/peers endpoints with exponential-backoff retries. If any peer returns a non-empty follower list, an existing Raft cluster is detected and this node stands down.

// In ClusterManager.java — the safety check before bootstrapping
private boolean isCandidateNode(List<DiscoveryNode> candidateNodes) {
    // Only the first node in the sorted list may bootstrap
    Optional<DiscoveryNode> firstNode = candidateNodes.stream().findFirst();
    String localNodeId = clusterService.getDiscovery().getLocalInstanceInfo().split(":")[0];

    if (!firstNode.map(n -> n.getNodeId().equals(localNodeId)).orElse(false)) {
        return false; // Not first: wait for leader contact
    }

    // First node: probe all others for an existing cluster before starting
    boolean clusterExists = detectExistingCluster(otherCandidates);
    return !clusterExists; // Only bootstrap if no existing cluster found
}

Non-candidate nodes start with an empty Raft group — RaftGroup.valueOf(raftGroupId, List.of()). This is intentional and follows Ratis's membership-change protocol. An empty-group start means the node participates in no configuration until the leader explicitly adds it via setConfiguration().

Adding Nodes: The AddListenerTask

After the initial cluster forms, all subsequent membership changes are driven by a background task called AddListenerTask. It runs on the leader every 10 seconds and implements the following decision tree:

AddListenerTask runs (leader only)
        │
        ├── Are any followers unreachable?
        │       │
        │       ├── YES: Is majority quorum still intact?
        │       │       ├── NO  → Log error, do nothing (unsafe to reconfigure)
        │       │       └── YES → Promote available listeners to followers (if flag enabled)
        │       │                 OR find a spare node, POST /ratis/v1/start on it,
        │       │                 then call setConfiguration() adding it as a follower
        │       │
        │       └── NO: Are any listeners unreachable?
        │               ├── YES → Replace with a spare node as listener
        │               └── NO  → Is follower count < expectedMemberSize?
        │                           └── YES → Add spare node as follower
        │                               NO  → Is listener count < maxListeners?
        │                                       └── YES → Add spare node as listener
        │                                           NO  → Cluster at capacity, do nothing

The quorum math is deliberately based on expectedMemberSize (from config), not the current number of running followers. This is a Raft safety requirement: the quorum threshold must be stable, not a moving target that changes every time a node disappears.

// In AddListenerTask.java — quorum safety check
int majorityQuorum = (expectedMemberSize / 2) + 1;

if (reachableFollowers.size() < majorityQuorum) {
    log.error("CRITICAL: Majority quorum lost! Reachable followers: {}, Required: {}",
            reachableFollowers.size(), majorityQuorum);
    // Do nothing — unsafe to change configuration without quorum
    return;
}

When a spare node is promoted, AtomDB uses a two-step process. First, it POSTs to the spare node's /ratis/v1/start endpoint to start the Raft server process on that node. Then it calls client.admin().setConfiguration() through the Raft protocol to officially add the peer to the group. The new node joins with an empty group and learns the current log from the leader through the normal Raft log replication mechanism.

The Peer String Format

All discovery configuration encodes nodes in a compact string format:

nodeId:host:raftPort:appPort:adminPort
n1:0.0.0.0:9000:8080:8081

• raftPort (9000): used by Ratis gRPC for log replication between peers

• appPort (8080): used by the leader's HTTP client to call /ratis/v1/start on spare nodes

• adminPort (8081): pinged during awaitDiscovery() to check liveness

Listeners: A Non-Voting Standby Mechanism

In Raft, a listener receives all log entries from the leader and maintains a current copy of the state, but it does not count toward the quorum. This has a useful operational property: a listener can be added to the cluster without changing the quorum size, making it a zero-risk standby.

AtomDB uses listeners as a first tier of high-availability:

With enablePromotionOfListenersToFollowers: false (the safe default), the leader will never silently swap a follower for a listener — any such promotion is a deliberate operator action. With it set to true, recovery from a single-follower failure becomes fully automatic, in a matter of seconds.

// In AddListenerTask.java — listener promotion gate
boolean enableListenerPromotion = clusterService.getClusterHealthStrategy()
        .isEnablePromotionOfListenersToFollowers();

if (enableListenerPromotion && !currentListeners.isEmpty()) {
    int promotionCount = Math.min(followersNeeded, currentListeners.size());
    for (int i = 0; i < promotionCount; i++) {
        RaftPeer listenerToPromote = currentListeners.get(i);
        RaftPeer promotedFollower = RaftPeer.newBuilder()
                .setId(listenerToPromote.getId())
                .setAddress(listenerToPromote.getAddress())
                .setStartupRole(RaftPeerRole.FOLLOWER)
                .build();
        currentFollowers.add(promotedFollower);
    }
}

A concrete scenario with a 5-node cluster (memberSize: 3, maxListeners: 1):

Initial state:   n1(follower) n2(follower) n3(follower) n4(listener) n5(spare, not started)

n2 goes down:
  AddListenerTask detects unreachable follower n2
  Quorum check: 2 reachable followers ≥ majorityQuorum(2) — safe to proceed
  enablePromotionOfListenersToFollowers: false → skip listener promotion
  Find spare node n5 → POST /ratis/v1/start to n5
  Call setConfiguration([n1, n3, n5], [n4])

Result:          n1(follower) n3(follower) n5(follower) n4(listener)
                 Quorum restored. n2 can rejoin later as a follower.

Snapshot Durability: Surviving a Full Cluster Restart

An in-memory state machine is convenient but fragile. If all nodes lose their disks simultaneously — or if a containerised deployment recycles all pods at once — every committed write is gone. To address this, AtomDB supports uploading snapshots to S3-compatible object storage.

Taking a Snapshot

A snapshot serialises the entire ConcurrentHashMap to a local file, computes its MD5, and then uploads both the snapshot and its checksum to S3 under a configurable prefix.

// In KeyValueStateMachine.java
@Override
public long takeSnapshot() {
    final TermIndex last = getLastAppliedTermIndex();

    // 1. Serialise the in-memory map to a local file
    File snapshotFile = s3Storage.getSnapshotFile(last.getTerm(), last.getIndex());
    try (ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream(snapshotFile))) {
        out.writeObject(new ConcurrentHashMap<>(keyValueStore));
    }

    // 2. Compute and save an MD5 checksum alongside the snapshot
    MD5Hash md5 = MD5FileUtil.computeAndSaveMd5ForFile(snapshotFile);

    // 3. Upload the snapshot file to S3 at the configured prefix
    s3Storage.uploadSnapshotToS3(last.getTerm(), last.getIndex(), snapshotFile);
    s3Storage.updateLatestSnapshot(new SingleFileSnapshotInfo(...));

    return last.getIndex();
}

The snapshot file name encodes the Raft term and index at which it was taken (e.g. snapshot.4_1892). On startup, loadSnapshot downloads the latest snapshot from S3 before the Raft log replay begins. The new node catches up to the snapshot's index instantly, and then only needs to replay the delta between the snapshot and the current log head — which is typically small.

# Minimal S3 storage config in your YAML
storageConfig:
  type: S3
  bucketName: "my-service-atomdb"
  region: "us-east-1"
  accessKey: "your-access-key"
  secretKey: "unused"                        # token-based auth
  endpoint: "https://your-s3-endpoint"
  pathStyleAccess: true
  snapshotPrefix: "my-service-snapshots-prod/"  # unique per service+env

The snapshotPrefix is the single most important knob to get right in production: use a unique value per service and per environment. Sharing a prefix between staging and production is the fastest path to a corrupted cluster after an accidental cross-environment snapshot download.

Integration Tests: Real Servers, Real Raft

Unit tests that mock Raft are not very useful — the interesting bugs live in timing, log replication races, and node failure/recovery sequences. AtomDB's integration tests spin up real in-process Dropwizard servers on dynamic ports using JUnit 5 extension.

@RegisterExtension
static MultiInstanceAtomDbExtension cluster = MultiInstanceAtomDbExtension.builder()
        .instanceCount(3)
        .quorumSize(3)
        .startupTimeout(Duration.ofSeconds(120))
        .build();

beforeAll calls ConfigGenerator.generateClusterConfigs() to produce dynamic-port YAML configs, starts each instance sequentially with a 3-second gap (to avoid bootstrap races), and then polls ClusterHealthChecker.waitForAllPeersInCluster() until the expected number of followers are visible in the cluster's listPeers() response. Only then does the first test method run.

Verifying Replication

The most fundamental correctness property: a PUT on node 1 must be visible on node 2.

@Test
void testPutOnOneNodeGetOnAnother() {
    AtomDbClient client1 = cluster.getClient("n1");
    AtomDbClient client2 = cluster.getClient("n2");

    AtomDbResponse<String> putResponse = client1.put("replication-key", "replication-value");
    assertThat(putResponse.isSuccess()).isTrue();

    cluster.waitForReplication();

    AtomDbResponse<String> getResponse = client2.getKey("replication-key");
    assertThat(getResponse.isSuccess()).isTrue();
    assertThat(getResponse.getResponse()).isEqualTo("replication-value");
}

Leader Failover

LeaderFailoverTest validates the complete failover lifecycle: stop the leader, verify the remaining two nodes elect a new leader, write new data, restart the former leader, and confirm it rejoins as a follower and catches up.

void testLeaderFailoverAndRecovery() throws Exception {
    verifyInitialClusterFormation();    // all 3 nodes up, listPeers returns 3 followers
    testClusterBeforeFailover();        // PUT/GET works
    stopLeaderNode();                   // cluster.stopNode("n1")
    testClusterAfterLeaderFailure();    // PUT/GET still works via new leader (n2 or n3)
    restartFormerLeaderNode();          // cluster.startNode("n1")
    verifyNodeRejoinsCluster();         // listPeers shows n1 as follower again
    testClusterAfterRecovery();         // all 3 nodes serve reads/writes
}

Follower Replacement

FollowerRecoveryTest uses a 5-node cluster (3 followers, 1 listener, 1 spare) and asserts that when a follower dies, the spare is automatically promoted within the AddListenerTask cycle:

@RegisterExtension
static MultiInstanceAtomDbExtension cluster = MultiInstanceAtomDbExtension.builder()
        .instanceCount(5)
        .quorumSize(3)
        .expectedPeerCount(4) // n1-n4 join; n5 remains spare until needed
        .startupTimeout(Duration.ofSeconds(180))
        .build();

After killFollowerAndVerifyPromotion() stops node n2, the test uses Awaitility to poll listPeers() until it sees exactly 3 followers again — with n5 promoted into n2's slot.

Listener Synchronisation

ClusterListenerSyncTest proves the non-voting-member guarantee: a listener keeps a current replica of the key-value store but its failure does not affect the voting quorum. The test simultaneously kills the listener node and one quorum follower, verifying that the remaining 2 quorum nodes still accept writes and serve reads.

Integrating as a Client

If you are writing a Java service that wants to talk to an AtomDB cluster without embedding the full Raft server (for example, a service that is not part of the cluster but needs to read configuration from it), the atomdb-client module provides a thin Feign interface:

<dependency>
    <groupId>com.snehasishroy</groupId>
    <artifactId>atomdb-client</artifactId>
    <version>1.0.0</version>
</dependency>

AtomDbClient client = Feign.builder()
        .client(new OkHttpClient())
        .encoder(new PlainTextEncoder(new JacksonEncoder()))
        .decoder(new JacksonDecoder())
        .target(AtomDbClient.class, "http://atomdb-node1:8080/");

// Store a value
AtomDbResponse<String> putResp = client.put("feature.flag.rollout", "true");
assertThat(putResp.isSuccess()).isTrue();

// Read it back
AtomDbResponse<String> getResp = client.getKey("feature.flag.rollout");
System.out.println(getResp.getResponse()); // "true"

The client interface itself is minimal by design:

public interface AtomDbClient {
    @RequestLine("GET /kv/v1/{key}")
    AtomDbResponse<String> getKey(@Param("key") String key);

    @RequestLine("PUT /kv/v1/{key}")
    @Headers("Content-Type: text/plain")
    AtomDbResponse<String> put(@Param("key") String key, String value);

    @RequestLine("GET /cluster/v1/peers")
    AtomDbResponse<ClusterPeersResponse> listPeers();

    @RequestLine("POST /snapshot/v1/")
    void triggerSnapshot();
}

Because reads are served by the leader and the leader can change during a failover, it is good practice to point your Feign target at a load balancer (or use retry logic with multiple target URLs) rather than hard-coding a single node.

Discovery Strategies: From Local Dev to Production

AtomDB supports three discovery modes to cover the full lifecycle from laptop to production:

In DYNAMIC mode every discoverable node — including future spares — must appear in the peers list upfront. A node not listed can never be discovered by the AddListenerTask.

In DROVE mode the control plane queries the Drove orchestration API (https://github.com/PhonePe/drove-orchestrator) to discover all healthy instances for the service. This is the zero-configuration production path - deploy a new instance, Drove registers it, AtomDB discovers it on the next AddListenerTask tick.

# Production DROVE config
atomDBBundleConfig:
  clusterConfiguration:
    discoveryConfig:
      type: DROVE
      droveEndpoint: ${DROVE_ENDPOINT_URL}
      raftPortName: raft
      communicationPortName: main
    clusterHealthConfig:
      type: DYNAMIC
      memberSize: 3
      maxListeners: 1
      enablePromotionOfListenersToFollowers: false
    groupUUID: 02511d47-d67c-49a3-9011-abb3109a44c2

Conclusion

AtomDB started as an experiment: what is the minimum viable design for a strongly-consistent, embedded key-value store that a team could adopt with focus on minimalistic external dependency? The answer turned out to be a surprisingly thin layer on top of Apache Ratis.

The core insight is the separation of concerns between Raft (the protocol) and the control plane (the operational intelligence). Ratis handles log replication, leader election, and membership changes — but it does not decide when to add nodes, which nodes to promote, or how to bootstrap a fresh cluster. That reasoning lives in ClusterManager and AddListenerTask, and it is the part of the system that is most specific to AtomDB's operational model.

The result is a system where you can:

• PUT a key on any node, get a linearisable write-after-quorum guarantee

• GET a key and always read the latest committed value

• Lose a minority of nodes and have the cluster self-heal with spare nodes within seconds

• Survive a full cluster restart by downloading a snapshot from S3

• Write tests that start real Raft clusters in-process, exercise actual failover sequences, and clean up after themselves in a single JUnit lifecycle

The source code is available at github.com/snehasishroy/atomdb. If you embed it in a Dropwizard application and find a bug or a missing feature, pull requests are very welcome.

P.S: I have done extensive testing to make sure the first release does not have any known bugs. However, this has not been tested on production env yet. Thank you for sticking to the end.

Source

Will Simple Sharding help?

One way to isolate this problem is to create virtual shards i.e. isolate requests coming from clients so that they are served only by some specific instances. Generally this is done based on some hashing e.g. modulo / consistent.

In the below diagram, all the requests originating from client names starting from A or B goes to Worker1 and Worker2. So if Alpha request is a poison pill, only two servers will get impacted. Remaining servers will be unaffected as the requests never reach there.

Do note that the Bravo request won't be fulfilled either because both Worker1 and Worker2 have crashed/hung due to the poison pill request from Alpha. One dirty fish has poisoned the entire lake.

This strategy has definitely reduced the failure radius but still requests from Bravo are not being served. Can we do better?

Source

Can increasing the shards help?

The issue with the above was the way we created shards — there were simply too few combinations available — as each instance can be mapped to only one shard.

If we allow each instance to be mapped to multiple shards, then we can increase the number of combinations available and reduce our unavailability.

In the below diagram, we created 8 shards to map 8 clients. Previously we only had 4 shards.

Source

Note that each workers are mapped to multiple shards e.g. Worker2 is mapped to Shard1 and Shard2.

Now if requests from Alpha crashes/hungs up Worker1 and Worker2 — requests from Bravo would still continue to work as they are mapped to Worker2 and Worker3. Since Worker3 is still alive, requests from Bravo would still get served.

Let’s Shuffle!

Hope you were able to grasp the fundamentals using the above example. The idea is to create more shards with as few overlaps as possible. Let’s see how we can make a generic solution.

Given n application servers, if we randomly choose k instances and make them part of one virtual shard, then each shard would have k instances. The probability of 2 shards with 100% overlap would drastically go down as we increase the value of k.

To simplify, if we have 10 cards and we want to choose 4 cards among them, then there will be a total of 10 choose 4 combinations = 210 total combinations. If we randomly generate 2 such combinations, then the probability of them having 100% overlap, i.e., all 4 cards being the same, would be (1 / 210) ~= 0.47%.

Now to relate this analogy to our problem statement, given 10 application instances, if we randomly choose 4 instances to create a virtual shard, the probability of 2 shards with 100% overlap would be ~0.47%. This indicates that a poison request can only impact 0.47% of shards. We can further reduce this by increasing the total number of instances or increasing the shard size.

Talk is cheap, show me the code!

Shuffle sharding can be implemented in two ways — stateless or stateful.

Stateless, as the name indicates, does not persist any state data, i.e., shard information in a DB Store. It simply identifies the target application instances from an identifier. The target application instances are the members of the virtual shard mapped to that request.

Stateful sharding goes a bit further and persists the information of the shards to a database, which allows further customization of the way shards are created, e.g., customizing the shard assignment strategy by tuning weights.

Let’s see how we can implement stateless shuffle sharding using a simple strategy of generating multiple hashes from a unique identifier, followed by mapping that hash to a unique node.

public Set<Integer> assignNodes(String customerId) {
    Set<Integer> assignedNodes = new HashSet<>();

    // Need to find 4 nodes
    for (int i = 0; i < 4; i++) {
        // Create a unique input for each Node selection
        String hashInput = customerId + ":" + i;
        // maps a hash to a NodeId
        int nodeId = hashToNodeId(hashInput); 

        // Handle collisions by trying the next available Node
        while (assignedNodes.contains(NodeId)) {
            nodeId = (nodeId + 1) % config.getTotalNodes();
        }
        assignedNodes.add(nodeId);
    }
    return assignedNodes;
}

We can also leverage multiple hash functions, similar to a Bloom Filter, to generate multiple hashes instead of generating multiple inputs from the identifier, in case multiple unique inputs can't be generated.

Practical UseCases

Shuffle Sharding is a powerful and versatile technique used not only by AWS but also in popular open-source projects like Grafana Loki and Grafana Mimir.

In a recent AWS Tech Talk, I learned that shuffle sharding is extensively used in S3 to introduce decorrelation in the system.

Identify drive to store data

Shuffle Sharding helps randomly select a drive to store the data for your bucket, instead of directly mapping drives to buckets.

In order to ensure a balanced disk utilization, a very clever technique known as Two random choices is used.

Law of of two random choices states that randomly pick two drives and choose the one with the lower disk utilization. This helps in avoiding scanning of all the drives in the system and maintaining info regarding their disk utilization.

Resolve DNS Queries

Shuffle Sharding is also used to resolve DNS queries. For example, when you look up s3.amazonaws.com or mybucket.s3.amazonaws.com, it returns multiple answers to DNS queries.

Bucket requests can randomly go to any server. It doesn't matter which server the request goes to because the buckets aren't tied to any specific server.

Reduce Latencies

Shuffle Sharding is also used in AWS CRT (Common Base library across AWS offerings) to improve latencies. CRT dynamically tracks the latency distributions and cancels the request going beyond p95 latencies, and retries it so the request goes to another host. This is gambling but it has paid off.

Since S3 leverages Erasure Coding to store shards of the data, Shuffle Sharding is also used to reduce latencies by cancelling requests going to slow shards and retrying so it goes to another shard.

Thank you for reading. Hope you learnt something new. If you have any questions, please do comment.

Appendix

• https://catalog.workshops.aws/well-architected-reliability/en-US/4-failure-management/2-fault-isolation/10-fault-isolation-with-shuffle-sharding

• https://grafana.com/docs/loki/latest/operations/shuffle-sharding/

You’re seeing multiple identical requests hitting your database or API. They all ask for the same thing and they all make separate, expensive calls.

Sound familiar?

If you’ve ever wondered, “Why are we doing the same thing five times in parallel?”, you’re not alone. This is a classic performance anti-pattern — and request coalescing can fix it.

So, What’s the Problem?

Imagine multiple users open the same product page at the same time. Or dozens of threads try to load the same user profile. They all hit your service with the same request—say, getUser("123")—within milliseconds of each other.

Now here’s the kicker: instead of sharing the work, each thread fires off its own request to the database or a remote API.

Why? Because your service has no idea that others are doing the same thing.

Let’s break it down:

Thread A: fetch("user123") → starts DB/API call
Thread B: fetch("user123") → starts DB/API call
Thread C: fetch("user123") → starts DB/API call

That’s three expensive calls… for the same data.

The Thundering Herd Problem

This scenario becomes even worse when a cache expires or a cold-start occurs. Suddenly, thousands of requests for the same key hit your backend simultaneously. This is known as the thundering herd problem.

In systems with shared caching or batch jobs, thundering herds can:

• Spike traffic to your database or upstream API

• Cause rate-limiting, timeouts, or failures

• Lead to cascading issues in downstream services

Why does this happen? Because each thread/client sees a cache miss and rushes to fetch the data independently—without knowing others are doing the same.

Can We Do Better?

What if you could say:

“Hey, I see someone is already fetching this. Let me just wait and use their result.”

That’s what request coalescing is all about.

What Is Request Coalescing?

Request coalescing is a technique where multiple concurrent requests for the same key are merged into one. Instead of all threads doing the same thing, only the first one does the work. The others just wait—and then reuse the result.

Here’s how it looks with coalescing:

Thread A: fetch("user123") → starts DB/API call
Thread B: fetch("user123") → waits for result from A
Thread C: fetch("user123") → waits for result from A

Only one call goes through. Everyone else benefits.

Where Should You Use It?

Request coalescing makes sense when

• The same key is requested often (e.g., trending topics, popular users).

• The backend call is expensive.

• You’re dealing with cold cache or frequent expirations.

• You use TTL-based caching and care about stability.

Let's Build !

public class User {
    String name;

    public User(String name) {
        this.name = name;
    }
}

This is a simple POJO representing a User. In real systems, this might come from a database or remote API.

public class UserDao {
    // use synchronized to simulate a even higher load by allowing only one thread to go through
    public synchronized User fetchByName(String name) {
        // simulate db fetch which takes 0.5 sec
        Stopwatch started = Stopwatch.createStarted();
        LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(500));
        User user = new User(name);
        long elapsed = started.elapsed(TimeUnit.MILLISECONDS);
        log.info("Took {} ms to fetch user", elapsed);
        return user;
    }
}

This class simulates a costly database fetch:

• Uses synchronized to throttle concurrent access (imitating heavy load).

• Sleeps for 500ms to simulate latency.

This is important to see the benefit when coalescing kicks in — only one of these slow fetches should happen!

public class UserController {
    private final boolean isCoalescingEnabled;
    UserDao userDao;
    RequestCoalescer<User> requestCoalescer;

    UserController(UserDao userDao, boolean isCoalescingEnabled) {
        this.userDao = userDao;
        this.requestCoalescer = new RequestCoalescer<>();
        this.isCoalescingEnabled = isCoalescingEnabled;
    }

    public User lookupName(String name) {
        if (isCoalescingEnabled) {
            return requestCoalescer.subscribe(name, () -> userDao.fetchByName(name));
        } else {
            return userDao.fetchByName(name);
        }
    }
}

This class controls how requests are handled:

• You can toggle coalescing on/off using isCoalescingEnabled.

• If enabled, the controller delegates to the coalescer.

• If disabled, it simply hits the DAO each time — resulting in multiple slow, redundant fetches.

This makes it easier to benchmark and demonstrate the benefits of coalescing.

public class RequestCoalescer<T> {
    Map<String, CompletableFuture<T>> inFlightRequests = new ConcurrentHashMap<>();
    public T subscribe(String key, Supplier<T> supplier) {
        CompletableFuture<T> future = getOrCreateFuture(key, supplier);
        return future.join();
    }

    private CompletableFuture<T> getOrCreateFuture(String key, Supplier<T> supplier) {
        CompletableFuture<T> future = inFlightRequests.get(key);
        if (future != null) {
            return future;
        }
        CompletableFuture<T> newFuture = new CompletableFuture<>();
        CompletableFuture<T> oldFuture = inFlightRequests.putIfAbsent(key, newFuture);
        if (oldFuture != null) {
            return oldFuture;
        } else {
            CompletableFuture.supplyAsync(() -> {
                try {
                    T result = supplier.get();
                    newFuture.complete(result);
                    inFlightRequests.remove(key, newFuture);
                    return result;
                } catch (Exception e) {
                    newFuture.completeExceptionally(e);
                    inFlightRequests.remove(key, newFuture);
                    // return value is unused - newFuture is actually used.
                    return null;
                }
            });
            return newFuture;
        }
    }
}

Step 1: Check if a fetch for this key is already happening. If so, return the existing future.

Step 2: Try to insert a new future. If another thread beat us to it, we return their future instead.

Step 3: If we won the race, start the fetch in a new thread. Once it’s done, we:

• Complete the future

• Remove the entry from the map (This is important to prevent memory leaks and avoid returning stale data)

@Slf4j
public class UserControllerTest {
    @ParameterizedTest
    @ValueSource(booleans = {true, false})
    public void testLookupName(boolean isCoalescingEnabled) throws InterruptedException {
        UserController userController = new UserController(new UserDao(), isCoalescingEnabled);
        CountDownLatch latch = new CountDownLatch(10);
        Stopwatch timer = Stopwatch.createStarted();
        for (int i = 0; i < 10; i++) {
            CompletableFuture.runAsync(() -> {
                userController.lookupName("test");
                latch.countDown();
            });
        }
        boolean await = latch.await(10, TimeUnit.SECONDS);
        Assertions.assertTrue(await);
        long seconds = timer.elapsed(TimeUnit.SECONDS);
        log.info("Took {} seconds", seconds);
        if (isCoalescingEnabled) {
            Assertions.assertTrue(seconds <= 1);
        } else {
            Assertions.assertTrue(seconds >= 5);
        }
    }
}

No code is completed until its tests are written.
In this test suite, we make 10 concurrent requests to lookup the user details for user with name test. When coalescing is disabled, the test takes 0.5 × 10 ~ 5 seconds to finish.

When coalescing is enabled, the test is finished in ~0.5 seconds because while the results of the first request is getting computed, the remaining requests are virtually short-circuited.

Gotchas

• Memory leaks: Always remove entries from the in-flight map after use.

• Timeouts: What if the fetch never finishes? Add appropriate timeouts.

• Error sharing: If the request fails, make sure others don’t cache a bad result.

• Over-coalescing: Don’t block forever; design with concurrency limits.

Thank you for reading. Hope you learnt something new today.

Fundamentals

Let's start with the basics. Kafka uses a custom binary protocol for sending and receiving messages.

The specifications define the request header as follows:

Specs defines the data types as

Here's an example of a request message:

00 00 00 23  // message_size:        35
00 12        // request_api_key:     18
00 04        // request_api_version: 4
6f 7f c6 61  // correlation_id:      1870644833
...

Every Kafka request is an API call. The Kafka protocol defines over 70 different APIs, all of which do different things. Here are some examples:

• Produce writes events to partitions.

• CreateTopics creates new topics.

• ApiVersions returns the broker's supported API versions.

A Kafka request specifies the API its calling by using request_api_key header field.

Message body

The schemas for the request and response bodies are determined by the API being called.

For example, here are some of the fields that the Produce request body contains:

• The name of the topic to write to.

• The key of the partition to write to.

• The event data to write.

On the other hand, the Produce response body contains a response code for each event. These response codes indicate if the writes succeeded.

As a reminder, requests and responses both have the following format:

1. message_size

2. Header

3. Body

API versioning

Each API supports multiple versions, to allow for different schemas. Here's how API versioning works:

• Requests use the header field request_api_version to specify the API version being requested.

• Responses always use the same API version as the request. For example, a Produce Request (Version: 3) will always get a Produce Response (Version: 3) back.

• Each API's version history is independent. So, different APIs with the same version are unrelated. For example, Produce Request (Version: 10) is not related to Fetch Request (Version: 10).

The ApiVersions API

The ApiVersions API returns the broker's supported API versions. For example, ApiVersions may say that the broker supports Produce versions 5 to 11, Fetch versions 0 to 3, etc.

Visualizing the Binary Protocol

Here is a great link that can help you visualize the binary protocol

Hands-on

So let’s build a POC of the Kafka server.

Let’s start our Server

public static void main(String[] args) {
    int port = 9092;
    try (ServerSocket server = new ServerSocket(port)) {
        server.setReuseAddress(true);
        log.info("Server started on port {}", port);
        while (true) {
            Socket client = server.accept();
            log.info("New client connected");
            handleClientAsync(client);
        }
    } catch (IOException e) {
        log.error("IOException: ", e);
    }
}

The server starts on port 9092 (the default Kafka port) and enters an infinite loop that waits for client connections. When a client connects, it passes the client socket to handleClientAsync(). The setReuseAddress(true) prevents "address already in use" errors when restarting the server.

Handle Multiple Clients Asynchronously

private static void handleClientAsync(Socket client) throws IOException {
    new Thread(() -> {
        try {
            processMessage(client);
        } catch (IOException e) {
            log.error("Error while handling client: ", e);
        } finally {
            try {
                client.close();
            } catch (IOException e) {
                log.error("Error closing client socket: ", e);
            }
        }
    }).start();
}

This method creates a new thread for each client connection, allowing the server to handle multiple clients simultaneously. It delegates the message handling to processMessage() This is a trivial implementation and will not scale well because of lack of Thread Pool.

Processing Client Messages

private static void processMessage(Socket client) throws IOException {
    DataInputStream in = new DataInputStream(client.getInputStream());
    DataOutputStream out = new DataOutputStream(client.getOutputStream());
    while (!client.isClosed()) {
        // Read the message header
        byte[] messageSize = new byte[4];
        int read = in.read(messageSize);
        if (read < 4) {
            log.info("Read fewer characters than expected: {}", read);
            break;
        }

        // Read protocol metadata
        byte[] apiKey = new byte[2];
        in.readFully(apiKey);
        byte[] apiVersion = new byte[2];
        in.readFully(apiVersion);
        byte[] correlationId = new byte[4];
        in.readFully(correlationId);

        // Read client identification
        byte[] clientIdLength = new byte[2];
        in.readFully(clientIdLength);
        byte[] clientId = new byte[ByteBuffer.wrap(clientIdLength).getShort()];
        in.readFully(clientId);

        log.info("Request from clientID {} for apiKey {} apiVersion {} correlationId {}", 
                new String(clientId), new String(apiKey), new String(apiVersion), new String(correlationId));

This is where we start seeing the protocol details. The method reads a message header containing size information, API key and version, correlation ID, and client ID. The use of DataInputStream helps in precise byte reading the binary protocol with fixed-size fields.

Handle DescribeTopicPartitions Request

        byte[] topicsArrayLength = null;
        byte[] topicNameLength = null;
        byte[] topicName = null;
        if (ByteBuffer.wrap(apiKey).getShort() == 75) {
            log.info("Received DescribeTopicPartitions request");
            byte[] tagBufferLength = new byte[1];
            in.readFully(tagBufferLength);
            log.info("Tag Buffer Length: {} ", ByteBuffer.wrap(tagBufferLength).get());

            topicsArrayLength = new byte[1];
            in.readFully(topicsArrayLength);
            log.info("Topics Array Length: {}", ByteBuffer.wrap(topicsArrayLength).get());

            topicNameLength = new byte[1];
            in.readFully(topicNameLength);
            log.info("Topic Name Length: {}", ByteBuffer.wrap(topicNameLength).get());

            topicName = new byte[ByteBuffer.wrap(topicNameLength).get() - 1];
            in.readFully(topicName);
            log.info("Topic Name: {}", new String(topicName));

            // Read additional fields
            byte[] tagBufferLength2 = new byte[1];
            in.readFully(tagBufferLength2);
            log.info("Tag Buffer2 Length: {}", ByteBuffer.wrap(tagBufferLength2).get());

            byte[] responsePartitionLimit = new byte[4];
            in.readFully(responsePartitionLimit);
            log.info("Response Partition Limit: {}", ByteBuffer.wrap(responsePartitionLimit).getInt());

            byte[] cursor = new byte[1];
            in.readFully(cursor);
            log.info("Cursor: {}", ByteBuffer.wrap(cursor).get());
        }

This block handles a specific request type - API key 75, which is DescribeTopicPartitions command. It reads several fields including the topic name, buffer lengths, and cursor information.

Sending a Response

        log.info("Remaining bytes in input stream: {}", in.available());
        in.skip(in.available());

        // Handle the request and send a response
        ByteBuffer responseBuffer = createResponseBuffer(apiKey, apiVersion, correlationId, 
                                                        topicsArrayLength, topicNameLength, topicName);

        out.write(responseBuffer.array(), 0, responseBuffer.position());
        out.flush();
        log.info("Response sent to client");
    }
}

After parsing the request, the code skips any remaining bytes (a defensive practice) and constructs a response using the createResponseBuffer() method. The response is then sent back to the client. This completes one request-response cycle in the continuous communication loop.

Crafting the Response using ByteBuffer

If you are interested in the Json reference, the official kafka client has one.

private static ByteBuffer createResponseBuffer(byte[] apiKey, byte[] apiVersion, byte[] correlationID, 
                                               byte[] topicsArrayLength, byte[] topicNameLength, byte[] topicName) {
    log.info("Creating response buffer");
    ByteBuffer responseBuffer = ByteBuffer.allocate(1024);
    responseBuffer.putInt(0); // Placeholder for message length
    responseBuffer.put(correlationID);

    // If API Key == 0x4b (75) (DescribeTopicPartitions)
    if (ByteBuffer.wrap(apiKey).getShort() == 75) {
        // Create DescribeTopicPartitions response
        responseBuffer.put((byte) 0); // Tag Buffer
        responseBuffer.putInt(0); // Throttle time
        responseBuffer.put(topicsArrayLength);// Topic array length
        responseBuffer.putShort((short) 3); // Error code
        responseBuffer.put(topicNameLength); // Topic name length
        responseBuffer.put(topicName); // Topic name
        responseBuffer.put(new byte[16]); // 16-byte null ID
        responseBuffer.put((byte) 0); // IsInternal == 0
        responseBuffer.put((byte) 1); // partition count + 1
        responseBuffer.putInt(0x00000DF8); // TopicAuthorizedOperations
        responseBuffer.put((byte) 0); // compact-encoded empty TAG_BUFFER

        responseBuffer.put((byte) 0xff); // Cursor
        responseBuffer.put((byte) 0); // Tag Buffer
    } else {
        // Create API versions response
        short apiVersionValue = ByteBuffer.wrap(apiVersion).getShort();
        short errorCode = (apiVersionValue < 0 || apiVersionValue > 4) ? (short) 35 : (short) 0;
        responseBuffer.putShort(errorCode);

        responseBuffer.put((byte) 3);
        // First API
        responseBuffer.putShort((short) 18); // API Versions 18
        responseBuffer.putShort((short) 0); // Min version
        responseBuffer.putShort((short) 4); // Max version
        responseBuffer.put((byte) 0); // Tagged fields for this API (compact encoded 0)

        // Second API
        responseBuffer.putShort((short) 75); // DescribeTopicPartitions 75
        responseBuffer.putShort((short) 0); // Min version
        responseBuffer.putShort((short) 0); // Max version
        responseBuffer.put((byte) 0); // Tagged fields for this API (compact encoded 0)

        responseBuffer.putInt(0); // Throttle time
        responseBuffer.put((byte) 0); // No tagged fields
    }

    // Update the message length at the beginning of the buffer
    int messageLength = responseBuffer.position() - 4;
    log.info("Message length: {}", messageLength);
    responseBuffer.putInt(0, messageLength);
    return responseBuffer;
}

This final method builds the response message. It branches based on the API key to create either a DescribeTopicPartitions response or the APIVersions response. The message length is calculated and inserted at the beginning of the buffer, a common pattern in binary protocols.

Verification

echo -n "00000031004b0000589eecfb000c6b61666b612d746573746572000212756e6b6e6f776e2d746f7069632d73617a0000000001ff00" \
| xxd -r -p | nc 192.168.1.6 9092 | hexdump -C

00000000  00 00 00 37 58 9e ec fb  00 00 00 00 00 02 00 03  |...7X...........|
00000010  12 75 6e 6b 6e 6f 77 6e  2d 74 6f 70 69 63 2d 73  |.unknown-topic-s|
00000020  61 7a 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |az..............|
00000030  00 00 00 01 00 00 0d f8  00 ff 00                 |...........|
0000003b

Hexdump of sent "DescribeTopicPartitions" request:
Idx  | Hex                                             | ASCII
-----+-------------------------------------------------+-----------------
0000 | 00 00 00 31 00 4b 00 00 58 9e ec fb 00 0c 6b 61 | ...1.K..X.....ka
0010 | 66 6b 61 2d 74 65 73 74 65 72 00 02 12 75 6e 6b | fka-tester...unk
0020 | 6e 6f 77 6e 2d 74 6f 70 69 63 2d 73 61 7a 00 00 | nown-topic-saz..
0030 | 00 00 01 ff 00                                  | .....

Hexdump of received "DescribeTopicPartitions" response:
Idx  | Hex                                             | ASCII
-----+-------------------------------------------------+-----------------
0000 | 00 00 00 37 58 9e ec fb 00 00 00 00 00 02 00 03 | ...7X...........
0010 | 12 75 6e 6b 6e 6f 77 6e 2d 74 6f 70 69 63 2d 73 | .unknown-topic-s
0020 | 61 7a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | az..............
0030 | 00 00 00 01 00 00 0d f8 00 ff 00                | ...........

.ResponseHeader
- .correlation_id (1486810363)
- .TAG_BUFFER
.ResponseBody
- .throttle_time_ms (0)
- .topic.length (1)
- .Topics[0]
  - .error_code (3)
  - .name (unknown-topic-saz)
  - .topic_id (00000000-0000-0000-0000-000000000000)
  - .is_internal (false)
  - .num_partitions (0)
  - .topic_authorized_operations (3576)
  - .TAG_BUFFER
- .next_cursor (null)
- .TAG_BUFFER

Thank you for reading. Hope you learnt something new.

Appendix

• https://binspec.org/kafka-api-versions-request-v4

• https://github.com/apache/kafka/blob/ce4940f9891a96819e54f8db097ce3824876e8e5/clients/src/main/resources/common/message/DescribeTopicPartitionsResponse.json

You have a task and a list of application instances that can execute it. After the task is completed, they must update the database with the result. How can we ensure that only one instance executes the task? We should focus on exactly-once execution of the task because each time the task runs it can have a side-effect — it might create a file, insert a row in the database, or perform an action that should happen only once.

Let's look at some of the options that are available to us.

Leader election

Perform a leader election amongst the instances using a consensus algorithm (e.g. Zookeeper or etcd) and let the leader execute the task.

Lock based Approach

Any instance can choose the task, but before executing it, a distributed lock must be obtained using a unique ID with an external system (e.g., Zookeeper, Redis, or Aerospike). If an instance gets the lock, no other instance has it, making it safe to execute the task. The other instances will either proceed to the next available task or wait a random amount of time before trying again to get the lock on the same task.

So you think you are done?

Gotchas

Let's dissect the edge cases with our previous approaches

Leader Election

Performing leader election using a consistent core approach, like Zookeeper, can still have edge cases during network partitions.

Imagine that initially, an instance named Alice was chosen as the leader. Due to a network issue or a GC pause, Alice was disconnected from the Zookeeper cluster long enough for the cluster to promote another instance, Bob, as the leader.

Bob picks up the same task and decides to execute it. However, Alice, the previous leader, is still active and decides to execute the same task, resulting in the task being executed twice.

Optimistic Approach

If the instance having the lock dies (Alice), no other instance can pick up the task. This can be solved by having a TTL (lease period) associated with the lock.

Choosing a TTL means the job must be completed within that time frame. If the TTL is too short, the lock might expire before the task is finished. If the TTL is too long, it takes more time to recover when an instance fails. For example, if Alice goes down, no other instance can get the lock until the lease expires.

There is still a possible issue where Alice thinks it has the lock, but its lease has expired and another instance, Bob, has taken the lock. This can occur if Alice is delayed by external factors (like GC pauses or page swaps) and its lease expires. When Alice resumes, it still believes it holds the lock and proceeds to execute the task.

Source

Based on the above examples, it's evident that we need another guardrail that can prevent a task from being executed twice.

Solution?

The problem with our earlier methods was the absence of safeguards during task execution. While a task was running, there were no checks to make sure it was safe to proceed.

What could some of these safeguards be?

Idempotency / Unique Key

If the side effect introduced during task execution has an idempotency or unique key associated with it, it becomes easier to detect whether something has already happened or is currently being executed.

If only the life were so simple.

Sometimes, it's not possible to have a unique key linked to the side effect. Why? Because it might be intentional. For example, in an HBase Cluster, the HBase master is in charge of assigning regions to a region server. If the HBase master believes a Region Server is down, it assigns that region (shard) to a new Region Server. Clients check the Region assignment mapping and redirect their read/write requests to the correct Region Server. Everything works fine as long as there is only one HBase master. In a split-brain scenario or network partitioning, two nodes might both think they are the Master and could perform Region reassignment tasks. This can severely affect data consistency. In this case, it's hard to assign a unique key to the region assignment task.

Fencing to the Rescue

Fencing is the process of isolating a cluster node or protecting shared resources when a node seems to be malfunctioning.

Previously, we relied on clients to behave correctly, assuming they wouldn't issue tasks unless they were the leaders. However, this assumption led to many edge cases in our design. To address this, we should add a check during task execution. Before running the task, the task execution service itself must verify if the caller has the authority to perform it.

How do we check this? The caller will provide a token that can be validated against the latest issued token (strong consistency) or the most recent token seen by the storage engine so far (linearizable reads/weak consistency).

Source

Let’s look at some practical examples which will help understand this concept as it can get quite tricky.

Split Brain Issue in Kafka Cluster

In a Kafka Cluster, there are multiple brokers — of which a single broker is selected as the controller of the entire cluster. Kafka uses Zookeeper as the consensus core. (It has now moved to Raft, but the fundamentals remain the same)

Source

What will happen if a Controller broker dies. One of the remaining brokers must get promoted as the Controller.

Keep in mind that you cannot truly know whether a broker has stopped for good or has experienced an intermittent failure. Nevertheless, the cluster has to move on and pick a new controller. This can lead to a zombie controller. A zombie controller can be defined as a controller node which had been deemed dead by the cluster and has come back online. Another broker has taken its place but the zombie controller might not know that yet.

This can easily happen. For example, if a nasty intermittent network partition happens or a controller has a long enough stop-the-world GC pause — the cluster will think it has died and pick a new controller. In the GC scenario, nothing has changed through the eyes of the original controller. The broker does not even know it was paused, much less that the cluster moved on without it. Because of that, it will continue acting as if it is the current controller. This is a common scenario in distributed systems and is called split-brain.

Let’s go through an example. Imagine the active controller really does go into a long stop-the-world GC pause. Its ZooKeeper session expires and /controller znode it registered is now deleted. Every other broker in the cluster is notified of this as they placed ZooKeeper Watches on it.

Source

To fix the controller-less cluster, every broker now tries to become the new controller itself. Let’s say Broker 2 won the race and became the new controller by creating the /controller znode first.

Every broker receives a notification that this znode was created and now knows who the latest leader is — Broker 2. Every broker except Broker 3, which is still in a GC pause. It is possible that this notification does not reach it for one reason or another (e.g OS has too many accepted connections awaiting processing and drops it). In the end, the information about the leadership change does not reach Broker 3.

Source

Broker 3’s garbage collection pause will eventually finish and it will wake up still thinking it is in charge. Remember, nothing has changed through its eyes.

Source

You now have two controllers which will be giving out potentially conflicting commands in parallel. This is something you do not want in your cluster. If not handled, it can result in critical failures.

If Broker 2 (new controller node) receives a request from Broker 3, how will it know whether Broker 3 is the newest controller or not? For all Broker 2 knows, the same GC pause might have happened to it too!

There needs to be a way to distinguish who the real, current controller of the cluster is.

There is such a way! It is done through the use of an epoch number (also called a fencing token). An epoch number is simply a monotonically increasing number — if the old leader had an epoch number of 1, the new one will have 2. Brokers can now easily differentiate the real controller by simply trusting the controller with the highest number. The controller with the highest number is surely the latest one, since the epoch number is always-increasing. This epoch number is stored in ZooKeeper.

Source

Here, Broker 1 stores the latest controllerEpoch it has seen and ignores all requests from controllers with a previous epoch number.

Let’s look at the Kafka code directly from the source

def registerControllerAndIncrementControllerEpoch(controllerId: Int): (Int, Int) = {
    val timestamp = time.milliseconds()

    // Read /controller_epoch to get the current controller epoch and zkVersion,
    // create /controller_epoch with initial value if not exists
    val (curEpoch, curEpochZkVersion) = getControllerEpoch
      .map(e => (e._1, e._2.getVersion))
      .getOrElse(maybeCreateControllerEpochZNode())

    // Create /controller and update /controller_epoch atomically
    val newControllerEpoch = curEpoch + 1
    val expectedControllerEpochZkVersion = curEpochZkVersion

    debug(s"Try to create ${ControllerZNode.path} and increment controller epoch to 
$newControllerEpoch with expected controller epoch zkVersion $expectedControllerEpochZkVersion")

    def checkControllerAndEpoch(): (Int, Int) = {
      val curControllerId = getControllerId.getOrElse(throw new ControllerMovedException(
        s"The ephemeral node at ${ControllerZNode.path} went away while checking 
whether the controller election succeeds. Aborting controller startup procedure"))
      if (controllerId == curControllerId) {
        val (epoch, stat) = getControllerEpoch.getOrElse(
          throw new IllegalStateException(s"${ControllerEpochZNode.path} existed before 
but goes away while trying to read it"))

        // If the epoch is the same as newControllerEpoch, it is safe to infer that the 
        // returned epoch zkVersion is associated with the current broker during 
        // controller election because we already knew that the zk
        // transaction succeeds based on the controller znode verification. Other rounds of controller
        // election will result in larger epoch number written in zk.
        if (epoch == newControllerEpoch)
          return (newControllerEpoch, stat.getVersion)
      }
      throw new ControllerMovedException("Controller moved to another broker. 
Aborting controller startup procedure")
    }

    def tryCreateControllerZNodeAndIncrementEpoch(): (Int, Int) = {
      val response = retryRequestUntilConnected(
        MultiRequest(Seq(
          CreateOp(ControllerZNode.path, ControllerZNode.encode(controllerId, timestamp), 
defaultAcls(ControllerZNode.path), CreateMode.EPHEMERAL),
          SetDataOp(ControllerEpochZNode.path, ControllerEpochZNode.encode(newControllerEpoch), 
expectedControllerEpochZkVersion)))
      )
      response.resultCode match {
        case Code.NODEEXISTS | Code.BADVERSION => checkControllerAndEpoch()
        case Code.OK =>
          val setDataResult = response.zkOpResults(1).rawOpResult.asInstanceOf[SetDataResult]
          (newControllerEpoch, setDataResult.getStat.getVersion)
        case code => throw KeeperException.create(code)
      }
    }

    tryCreateControllerZNodeAndIncrementEpoch()
  }

  private def maybeCreateControllerEpochZNode(): (Int, Int) = {
    createControllerEpochRaw(KafkaController.InitialControllerEpoch).resultCode match {
      case Code.OK =>
        info(s"Successfully created ${ControllerEpochZNode.path} with initial epoch ${KafkaController.InitialControllerEpoch}")
        (KafkaController.InitialControllerEpoch, KafkaController.InitialControllerEpochZkVersion)
      case Code.NODEEXISTS =>
        val (epoch, stat) = getControllerEpoch.getOrElse(throw new IllegalStateException(s"${ControllerEpochZNode.path} existed before but goes away while trying to read it"))
        (epoch, stat.getVersion)
      case code =>
        throw KeeperException.create(code)
    }


  def getControllerEpoch: Option[(Int, Stat)] = {
    val getDataRequest = GetDataRequest(ControllerEpochZNode.path)
    val getDataResponse = retryRequestUntilConnected(getDataRequest)
    getDataResponse.resultCode match {
      case Code.OK =>
        val epoch = ControllerEpochZNode.decode(getDataResponse.data)
        Option(epoch, getDataResponse.stat)
      case Code.NONODE => None
      case _ => throw getDataResponse.resultException.get
    }
  }

Key Implementation Details

Atomic Operations

tryCreateControllerZNodeAndIncrementEpoch() method handles two critical operations atomically using ZooKeeper's MultiRequest to execute both operations atomically. Either both succeed or both fail.

1. Creating a controller znode indicating this broker is the controller CreateOp()

2. Incrementing the controller epoch to prevent split-brain scenarios SetDataOp()

Optimistic Concurrency

SetDataOp() in tryCreateControllerZNodeAndIncrementEpoch() uses the ZooKeeper version number (expectedControllerEpochZkVersion) to ensure that the epoch is only updated if no one else has modified it since it was read - this is a classic example of optimistic updates.

Ephemeral Nodes

The controller znode is created as an ephemeral node (CreateMode.EPHEMERAL), which means it will be automatically deleted if the controller broker's ZooKeeper session expires - critical for automatic failure detection.

Takeaways

If you have read so far, I appreciate your patience. Hope you learnt something new today and Thank you for reading.

To support exactly once execution — performing Leader Election or taking a locks isn’t enough — there should be additional checks at the time of task execution to detect whether it’s safe to actually execute the task. Fencing Token is one such way — however generating one isn’t so straightforward.

Please feel free to ask any questions you might have in the comments.

References

• https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html

• https://hackernoon.com/apache-kafkas-distributed-system-firefighter-the-controller-broker-1afca1eae302

• Kafka Source Code

We want to create a Task Execution Service that can execute tasks in a fault-tolerant manner. The service should dynamically discover workers and assign tasks to them. If a worker dies while executing a task, the service should be able to find that a worker has died (or stopped responding) and reassign the task to a new worker, providing at least once guarantee for job execution. All these should be done in a highly scalable and distributed manner.

This will be a hands-on guide on implementing a distributed job execution service - so get your coffee mug ready.

Architecture

• Clients submit job details to Zookeeper and listen to the status updates again via Zookeeper.

• Multiple worker instances utilizes Zookeeper to perform leader election.

• The leader instances watches job path to listen for upcoming job and assigns the jobs to the available workers.

• The worker instances watches their assignment mapping path. When a new job is found, it gets executed and the completion status is updated.

• The client instances gets notified upon task completion by Zookeeper.

Zookeeper

Zookeeper is a robust service that aims to deliver coordination among distributed systems. It's widely used in open-source projects like Kafka and HBase as the central coordination service.

We will use CuratorFramework in our project as it provides high-level API's for interacting with Zookeeper.

Implementation Details

Let's look at the ClientResource - which provides a facade for task submission.

@Slf4j
@Path("/v1/client")
public class Client {
  private final ClientService clientService;

  @Inject
  public Client(CuratorFramework curator) {
    this.clientService = new ClientService(curator);
  }

  @POST
  public String createSumTask(@QueryParam("first") int a, @QueryParam("second") int b) {
    Runnable jobDetail =
        (Runnable & Serializable)
            (() -> System.out.println("Sum of " + a + " and " + b + " is " + (a + b)));
    return clientService.registerJob(jobDetail);
  }
}

The above code, allows clients to submit a sample runnable task that computes the sum of two numbers and prints it - this provides an easy way for input via Swagger. But the design is extensible - the client can submit any instance of the Runnable as a Job.

Now let's look at the ClientService

public class ClientService {
  private final CuratorFramework curator;

  public ClientService(CuratorFramework curator) {
    this.curator = curator;
  }

  public String registerJob(Runnable jobDetail) {
    String jobId = UUID.randomUUID().toString();
    syncCreate(ZKUtils.getJobsPath() + "/" + jobId, jobDetail);
    return jobId;
  }

  private void syncCreate(String path, Runnable runnable) {
    // create the ZNode along with the Runnable instance as data
    try {
      ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
      ObjectOutputStream objectOutputStream = new ObjectOutputStream(byteArrayOutputStream);
      objectOutputStream.writeObject(runnable);
      curator
          .create()
          .idempotent()
          .withMode(CreateMode.PERSISTENT)
          .forPath(path, byteArrayOutputStream.toByteArray());
    } catch (Exception e) {
      log.error("Unable to create {}", path, e);
      throw new RuntimeException(e);
    }
  }
}

Once a job is registered, a unique ID is assigned to it and a Persistent node is registered on the Zookeeper with the randomly generated job ID in the path e.g. /jobs/{job-id}. Do notice that the runnable is serialized to a byte array and stored in the ZNode directly.

Notice that we are creating the ZNode synchronously i.e. the function syncCreate will block until the ZNode is not created. In the later section, you will notice that we have used asynchronous operations to improve throughput.

Why are we creating paths? So that we can set up watches on it. Watches allow us to be notified of any changes under the watched path. Zookeeper will invoke the JobsListener whenever a new node is created or destroyed under the /jobs path.

public class JobsListener implements CuratorCacheListener {
  private final CuratorFramework curator;
  private final CuratorCache workersCache;
  private final ExecutorService executorService;
  private final WorkerPickerStrategy workerPickerStrategy;

  public JobsListener(
      CuratorFramework curator,
      CuratorCache workersCache,
      WorkerPickerStrategy workerPickerStrategy) {
    this.curator = curator;
    this.workersCache = workersCache;
    executorService = Executors.newSingleThreadExecutor();
    this.workerPickerStrategy = workerPickerStrategy;
  }

  @Override
  public void event(Type type, ChildData oldData, ChildData data) {
    if (type == Type.NODE_CREATED && data.getPath().length() > ZKUtils.JOBS_ROOT.length()) {
      String jobID = ZKUtils.extractNode(data.getPath());
      log.info("found new job {}, passing it to executor service", jobID);
      // an executor service is used in order to avoid blocking the watcher thread as the job
      // execution can be time consuming
      // and we don't want to skip handling new jobs during that time
      executorService.submit(
          new JobAssigner(jobID, data.getData(), curator, workersCache, workerPickerStrategy));
    }
  }
}

When a new job is found, we hand over the Job ID to a different thread because we don't want to block the watcher thread.

We are setting up the watcher using CuratorCache - which will be explained later on.

JobAssigner

Once a job has been created, we need to execute it by finding an eligible worker based on a strategy. We can either choose a worker randomly or in a round-robin manner. Once a worker is chosen, we need to create an assignment between a JobID and a Worker ID - we do so by creating a Persistent ZNode on the path /assignments/{worker-id}/{job-id} . Once the assignment is created, we delete the /jobs/{job-id} path.

public class JobAssigner implements Runnable {

  private final CuratorFramework curator;
  private final String jobID;
  private final CuratorCache workersCache;
  private final WorkerPickerStrategy workerPickerStrategy;
  private final byte[] jobData;
  private String workerName;

  public JobAssigner(
      String jobID,
      byte[] jobData,
      CuratorFramework curator,
      CuratorCache workersCache,
      WorkerPickerStrategy workerPickerStrategy) {
    this.jobID = jobID;
    this.curator = curator;
    this.workersCache = workersCache;
    this.workerPickerStrategy = workerPickerStrategy;
    this.jobData = jobData;
  }

  @Override
  public void run() {
    // from the list of workers, pick a worker based on the provided strategy and assign the
    // incoming job to that worker
    List<ChildData> workers =
        workersCache.stream()
            .filter(childData -> (childData.getPath().length() > ZKUtils.WORKERS_ROOT.length()))
            .toList();
    ChildData chosenWorker = workerPickerStrategy.evaluate(workers);
    workerName = ZKUtils.extractNode(chosenWorker.getPath());
    log.info(
        "Found total workers {}, Chosen worker index {}, worker name {}",
        workers.size(),
        chosenWorker,
        workerName);
    asyncCreateAssignment();
  }

  private void asyncCreateAssignment() {
    try {
      curator
          .create()
          .idempotent()
          .withMode(CreateMode.PERSISTENT)
          .inBackground(
              new BackgroundCallback() {
                @Override
                public void processResult(CuratorFramework client, CuratorEvent event) {
                  switch (KeeperException.Code.get(event.getResultCode())) {
                    case OK -> {
                      log.info(
                          "Assignment created successfully for JobID {} with WorkerID {}",
                          jobID,
                          workerName);
                      log.info(
                          "Performing async deletion of {}", ZKUtils.getJobsPath() + "/" + jobID);
                      asyncDelete(ZKUtils.getJobsPath() + "/" + jobID);
                    }
                    case CONNECTIONLOSS -> {
                      log.error(
                          "Lost connection to ZK while creating {}, retrying", event.getPath());
                      asyncCreateAssignment();
                    }
                    case NODEEXISTS -> {
                      log.warn("Assignment already exists for path {}", event.getPath());
                    }
                    case NONODE -> {
                      log.error("Trying to create an assignment for a worker which does not exist {}", event);
                    }
                    default -> log.error("Unhandled event {} ", event);
                  }
                }
              })
          .forPath(ZKUtils.ASSIGNMENT_ROOT + "/" + workerName + "/" + jobID, jobData);
      // Storing the job data along with the assignment, so that the respective worker need not
      // perform an additional call to get the job details.
      // This also simplifies the design - because we can delete the /jobs/{jobID} path once the
      // assignment  is completed - indicating that if an entry is present under /jobs, it's
      // assignment is not yet done.
      // This makes the recovery/reconciliation process much easier. Once a leader is elected, it
      // has to only perform liveliness check for the existing assignments.
      // TODO: Use MultiOp to perform assignment and deletion atomically
    } catch (Exception e) {
      log.error("Error while creating assignment for {} with {}", jobID, workerName, e);
      throw new RuntimeException(e);
    }
  }

  private void asyncDelete(String path) {
    // delete the provided ZNode
    try {
      curator
          .delete()
          .idempotent()
          .guaranteed()
          .inBackground(
              new BackgroundCallback() {
                @Override
                public void processResult(CuratorFramework client, CuratorEvent event) {
                  switch (KeeperException.Code.get(event.getResultCode())) {
                    case OK -> {
                      log.info("Path deleted successfully {}", event.getPath());
                    }
                    case CONNECTIONLOSS -> {
                      log.info(
                          "Lost connection to ZK while deleting {}, retrying", event.getPath());
                      asyncDelete(event.getPath());
                    }
                    default -> log.error("Unhandled event {}", event);
                  }
                }
              })
          .forPath(path);
    } catch (Exception e) {
      log.error("Unable to delete {} due to ", path, e);
      throw new RuntimeException(e);
    }
  }
}

WorkerPickerStrategy

We are using Strategy pattern to dynamically change the way we can choose a worker at runtime. The important thing to notice is that we have used compare and swap as a way to perform optimistic locking for RoundRobinWorker .

public interface WorkerPickerStrategy {
  ChildData evaluate(List<ChildData> workers);
}

// choose workers based on random strategy
public class RandomWorker implements WorkerPickerStrategy {
  @Override
  public ChildData evaluate(List<ChildData> workers) {
    int chosenWorker = (int) (Math.random() * workers.size());
    return workers.get(chosenWorker);
  }
}

// choose workers based on round robin strategy
public class RoundRobinWorker implements WorkerPickerStrategy {
  AtomicInteger index =
      new AtomicInteger(0); // atomic because this will be accessed from multiple threads

  @Override
  public ChildData evaluate(List<ChildData> workers) {
    int chosenIndex;
    while (true) { // repeat this until compare and set operation is succeeded
      chosenIndex = index.get();
      int nextIndex = (chosenIndex + 1) < workers.size() ? (chosenIndex + 1) : 0;
      // in case of concurrent updates, this can fail, hence we have to retry until success
      if (index.compareAndSet(chosenIndex, nextIndex)) {
        break;
      }
    }
    return workers.get(chosenIndex);
  }
}

WorkerService

Since the JobHandler creates an assignment using ZNode of form /assignments/{worker-id}/{job-id} , if a worker has to listen to upcoming assignments, a watch needs to be set on the /assignments/{worker-id} path.

Once the worker service is notified of the new assignment, it fetches the job details, deserializes it to an instance of Runnable, and passes it to an ExecutorService which performs the execution.

Once the runnable has been executed, we chain the future by updating the status of the job id. The status of a job ID is reflected by asynchronously creating an entry in /status/{job-id} . Once the entry is created, we perform the last operation in this orchestra - deletion of the assignment mapping.

public class AssignmentListener implements CuratorCacheListener {
  private final CuratorFramework curator;
  private final ExecutorService executorService;

  public AssignmentListener(CuratorFramework curator) {
    this.curator = curator;
    this.executorService = Executors.newFixedThreadPool(10);
  }

  @Override
  public void event(Type type, ChildData oldData, ChildData data) {
    if (type == Type.NODE_CREATED) {
      if (data.getPath().indexOf('/', 1) == data.getPath().lastIndexOf('/')) {
        // This filters out the root path /assignment/{worker-id} which does not contains any job id
        return;
      }
      String jobId = data.getPath().substring(data.getPath().lastIndexOf('/') + 1);
      log.info("Assignment found for job id {}", jobId);

      try {
        byte[] bytes = data.getData();
        ObjectInputStream objectInputStream =
            new ObjectInputStream(new ByteArrayInputStream(bytes));
        Runnable jobDetail = (Runnable) objectInputStream.readObject();
        log.info("Deserialized the JobId {} to {}", jobId, jobDetail);
        CompletableFuture<Void> future = CompletableFuture.runAsync(jobDetail, executorService);
        // Actual execution of the job will be performed in a separate thread to avoid blocking of
        // watcher thread
        log.info("Job submitted for execution");
        // once the job has been executed, we need to ensure the assignment is deleted and the
        // status of job has been updated. Currently there is no guarantee that post the execution,
        // this cleanup happens.
        // TODO: Implement a daemon service which performs cleanup
        future.thenAcceptAsync(__ -> asyncCreate(jobId, data.getPath()), executorService);
      } catch (Exception e) {
        log.error("Unable to fetch data for job id {}", jobId, e);
      }
    }
  }

  private void asyncCreate(String jobId, String assignmentPath) {
    log.info("JobID {} has been executed, moving on to update its status", jobId);
    // create the ZNode, no need to set any data with this znode
    try {
      curator
          .create()
          .withTtl(ZKUtils.STATUS_TTL_MILLIS)
          .creatingParentsIfNeeded()
          .withMode(CreateMode.PERSISTENT_WITH_TTL)
          .inBackground(
              new BackgroundCallback() {
                @Override
                public void processResult(CuratorFramework client, CuratorEvent event) {
                  switch (KeeperException.Code.get(event.getResultCode())) {
                    case OK -> {
                      log.info("Status updated successfully {}", event.getPath());
                      log.info("Performing deletion of assignment path {}", assignmentPath);
                      asyncDelete(assignmentPath);
                    }
                    case CONNECTIONLOSS -> {
                      log.error(
                          "Lost connection to ZK while creating {}, retrying", event.getPath());
                      asyncCreate(jobId, assignmentPath);
                    }
                    case NODEEXISTS -> {
                      log.warn("Node already exists for path {}", event.getPath());
                    }
                    default -> log.error("Unhandled event {}", event);
                  }
                }
              })
          .forPath(ZKUtils.getStatusPath(jobId), "Completed".getBytes());
    } catch (Exception e) {
      log.error("Unable to create {} due to ", ZKUtils.getStatusPath(jobId), e);
      throw new RuntimeException(e);
    }
  }

  private void asyncDelete(String path) {
    // delete the provided ZNode
    try {
      curator
          .delete()
          .idempotent()
          .guaranteed()
          .inBackground(
              new BackgroundCallback() {
                @Override
                public void processResult(CuratorFramework client, CuratorEvent event) {
                  switch (KeeperException.Code.get(event.getResultCode())) {
                    case OK -> {
                      log.info("Path deleted successfully {}", event.getPath());
                    }
                    case CONNECTIONLOSS -> {
                      log.info(
                          "Lost connection to ZK while deleting {}, retrying", event.getPath());
                      asyncDelete(event.getPath());
                    }
                    default -> log.error("Unhandled event {}", event);
                  }
                }
              })
          .forPath(path);
    } catch (Exception e) {
      log.error("Unable to delete {} due to ", path, e);
      throw new RuntimeException(e);
    }
  }
}

WorkersListener

When a worker is lost due to network partition, or application shutdown, the leader instance is notified using a watcher event. The leader then looks at all the tasks that were assigned to the lost worker by iterating over the assignment mappings /assignment/{worker-id}/{job-id} .

All the tasks are then recreated by re-creating an entry in the /jobs/{job-id} . This recreation triggers the entire workflow from the start.

public class WorkersListener implements CuratorCacheListener {

  private final CuratorCache assignmentCache;
  private final CuratorFramework curator;

  public WorkersListener(CuratorCache assignmentCache, CuratorFramework curator) {
    this.assignmentCache = assignmentCache;
    this.curator = curator;
  }

  @Override
  public void event(Type type, ChildData oldData, ChildData data) {
    if (type == Type.NODE_CREATED) {
      log.info("New worker found {} ", data.getPath());
    } else if (type == Type.NODE_DELETED) {
      // notice we have to check oldData because data will be null
      log.info("Lost worker {}", oldData.getPath());
      String lostWorkerID = oldData.getPath().substring(oldData.getPath().lastIndexOf('/') + 1);
      // map of job ids -> job data, which was assigned to the lost worker
      Map<String, byte[]> assignableJobIds = new HashMap<>();
      assignmentCache.stream()
          .forEach(
              childData -> {
                String path = childData.getPath();
                int begin = path.indexOf('/') + 1;
                int end = path.indexOf('/', begin);
                String pathWorkerID = path.substring(begin, end);
                if (pathWorkerID.equals(lostWorkerID)) {
                  String jobID = path.substring(end + 1);
                  log.info("Found {} assigned to lost worker {}", jobID, lostWorkerID);
                  assignableJobIds.put(jobID, childData.getData());
                }
              });
      // Assuming atomic creation of assignment path and deletion of tasks path (using MultiOp), we
      // can safely assume that no entry exists under /jobs for the assigned tasks.
      // So we can simulate job creation by recreating an entry in the /jobs entry.
      assignableJobIds.forEach(
          (jobId, jobData) -> asyncCreateJob(ZKUtils.getJobsPath() + "/" + jobId, jobData));
    }
  }

  private void asyncCreateJob(String path, byte[] data) {
    try {
      curator
          .create()
          .idempotent()
          .withMode(CreateMode.PERSISTENT)
          .inBackground(
              new BackgroundCallback() {
                @Override
                public void processResult(CuratorFramework client, CuratorEvent event) {
                  switch (KeeperException.Code.get(event.getResultCode())) {
                    case OK -> {
                      log.info("Job repaired successfully for {}", path);
                    }
                    case CONNECTIONLOSS -> {
                      log.error(
                          "Lost connection to ZK while repairing job {}, retrying",
                          event.getPath());
                      asyncCreateJob(event.getPath(), (byte[]) event.getContext());
                    }
                    case NODEEXISTS -> {
                      log.warn("Job already exists for path {}", event.getPath());
                    }
                    default -> log.error("Unhandled event {}", event);
                  }
                }
              },
              data)
          .forPath(path, data);
    } catch (Exception e) {
      log.error("Error while repairing job {}", path, e);
      throw new RuntimeException(e);
    }
  }
}

WorkerService - The humble plumber

Throughout the article, you might have noticed that we talked about a leader instance performing some work but never explained it. So let's talk about what a leader instance is and how an instance becomes a leader.

When a worker instance comes up - it enqueues itself for a chance to become a leader. We perform leader elections using the Curator framework ensuring that only a single instance can become a leader amongst the members.

The leader is entrusted to perform critical actions like watching the /jobs/ path and the /workers/ path. The remaining instances do not set up watches on these paths because we want to ensure a task is assigned to only one worker instance. If multiple instances were trying to perform the assignment, it would be difficult to coordinate among them without taking a lock. This is where the Zookeeper comes in and acts as the trusty coordination service.

public class WorkerService implements LeaderSelectorListener, Closeable {
  public WorkerService(CuratorFramework curator, String path) {
    this.curator = curator;
    leaderSelector = new LeaderSelector(curator, path, this);
    // the selection for this instance doesn't start until the leader selector is started
    // leader selection is done in the background so this call to leaderSelector.start() returns
    // immediately
    leaderSelector.start();
    // this is important as it automatically handles failure scenarios i.e. starts leadership after
    // the reconnected state
    // https://www.mail-archive.com/user@curator.apache.org/msg00903.html
    leaderSelector.autoRequeue();
    setup();
    workerPickerStrategy = new RoundRobinWorker();
  }

  private void setup() {
    registerWorker();
    asyncCreate(ZKUtils.getJobsPath(), CreateMode.PERSISTENT, null);
    asyncCreate(ZKUtils.getAssignmentPath(name), CreateMode.PERSISTENT, null);
    asyncCreate(ZKUtils.STATUS_ROOT, CreateMode.PERSISTENT, null);
  }

  private void registerWorker() {
    if (registrationRequired.get()) {
      log.info("Attempting worker registration");
      name = UUID.randomUUID().toString();
      log.info("Generated a new random name to the worker {}", name);
      asyncCreate(ZKUtils.getWorkerPath(name), CreateMode.EPHEMERAL, registrationRequired);
      asyncCreate(ZKUtils.getAssignmentPath(name), CreateMode.PERSISTENT, null);
      watchAssignmentPath();
      // irrespective of whether this node is a leader or not, we need to watch the assignment path
    }
  }

  // only the leader worker will watch for incoming jobs and changes to available workers
  private void watchJobsAndWorkersPath() {
    // in case leadership is reacquired, repeat the setup of the watches
    workersCache = CuratorCache.build(curator, ZKUtils.WORKERS_ROOT);
    workersCache.start();
    log.info("Watching workers root path {}", ZKUtils.WORKERS_ROOT);
    workersListener = new WorkersListener(assignmentCache, curator);
    workersCache.listenable().addListener(workersListener);

    jobsCache = CuratorCache.build(curator, ZKUtils.JOBS_ROOT);
    log.info("Watching jobs root path {}", ZKUtils.getJobsPath());
    jobsCache.start();
    jobsListener = new JobsListener(curator, workersCache, workerPickerStrategy);
    jobsCache.listenable().addListener(jobsListener);
  }

  private void watchAssignmentPath() {
    // No need to check for null here because once a session is reconnected after a loss
    // we need to start the assignment listener on the new worker id
    assignmentCache = CuratorCache.build(curator, ZKUtils.getAssignmentPath(name));
    log.info("Watching {}", ZKUtils.getAssignmentPath(name));
    assignmentCache.start();
    assignmentListener = new AssignmentListener(curator);
    assignmentCache.listenable().addListener(assignmentListener);
  }

  @Override
  public void takeLeadership(CuratorFramework client) {
    // we are now the leader. This method should not return until we want to relinquish leadership,
    // which will only happen, if someone has signalled us to stop
    log.info("{} is now the leader", name);
    // only the leader should watch the jobs and workers path
    watchJobsAndWorkersPath();
    lock.lock();
    try {
      // sleep until signalled to stop
      while (!shouldStop.get()) {
        condition.await();
      }
      if (shouldStop.get()) {
        log.warn("{} is signalled to stop!", name);
        leaderSelector.close();
      }
    } catch (InterruptedException e) { // this is propagated from cancel leadership election
      log.error("Thread is interrupted, need to exit the leadership", e);
    } finally {
      // finally is called before the method return
      lock.unlock();
    }
  }

  @Override
  public void stateChanged(CuratorFramework client, ConnectionState newState) {
    if (newState == ConnectionState.RECONNECTED) {
      log.error("Reconnected to ZK, Received {}", newState);
      // no need to start the leadership again as it is auto requeued but worker re-registration is
      // still required which will create a ZNode in /workers and /assignments path
      registerWorker();
    } else if (newState == ConnectionState.LOST) {
      log.error("Connection suspended/lost to ZK, giving up leadership {}", newState);
      registrationRequired.set(true);
      // This is required as the assignment cache listens on the {worker id} which is ephemeral
      // In case of a lost session, it's guaranteed that the {worker id} would have expired
      // Once the session is reconnected, we need to set up the assignment listener again on a new
      // worker id
      // TODO: Figure out a way to simulate the disconnection from zookeeper only by one instance
      log.info("Removing the watcher set on the assignment listener");
      assignmentCache.listenable().removeListener(assignmentListener);
      assignmentCache.close();
      if (workersCache != null) {
        log.info("Removing the watcher set on the workers listener");
        workersCache.listenable().removeListener(workersListener);
        workersCache.close();
      }
      if (jobsCache != null) {
        log.info("Removing the watcher set on the jobs listener");
        jobsCache.listenable().removeListener(jobsListener);
        jobsCache.close();
      }
      // throwing this specific exception would cause the current thread to interrupt and would
      // cause and InterruptedException
      throw new CancelLeadershipException();
    } else if (newState == ConnectionState.SUSPENDED) {
      // https://stackoverflow.com/questions/41042798/how-to-handle-apache-curator-distributed-lock-loss-of-connection
      log.error("Connection has been suspended to ZK {}", newState);
      // TODO: After increasing the time out, verify whether no other instance gets the lock before
      // the connection is marked as LOST
    }
}

When we detect that our instance has lost its connection from Zookeeper, we remove any watches that have been set up and throw a CancelLeadershipException. And then we wait until we are reconnected to the Zookeeper.

Once reconnected, we generate a new name for the worker and set up appropriate watches. Since autoRequeue() was enabled during the leader election, the instance will enqueue itself for a chance of becoming a leader.

Conclusion

If you have read so far, I appreciate your patience. Hope you learnt something new today. Thank you for reading.

Please feel free to ask any questions you might have in the comments.

Appendix

• ZooKeeper watches are single-threaded.

• Link to the Code Repository

Preface

If you have ever missed your morning flight because your alarm never rang, you understand the importance of reliable alarms.

At PhonePe, we ensure that your alarms always ring at the time you want them to ring. Consider a scenario that happens daily at PhonePe - Merchant Settlements. If a merchant has performed multiple transactions during the day, at the end of the day, we want to ensure the final amount gets credited to their account. Any delay in the jobs getting triggered can cause a delay in the merchant settlement which can lead to a loss of customer trust.

Consider another scenario - Coupons invalidation. If you have received a coupon in PhonePe, you must have noticed that the coupon has a fixed validity. Now multiply this by hundreds of thousands or even millions of coupons. This necessitates a platform that can reliably schedule tasks at a predetermined schedule to invalidate all the coupons.

Patterns similar to this keep presenting themselves across different systems and at incredible scale.

The obvious approach would have been to use embedded schedulers, i.e. a scheduler service that runs within the client application and allows the client to schedule jobs in the future. However, this presents a set of challenges, especially the lack of fault tolerance. What will happen to the scheduled job in case the client application goes down? To solve this, we would need persistence, but it would need a lot of complex coordination across hundreds of stateless containers. Why? Because in most cases, we would not want the same job to be executed by multiple containers. Then there is the issue of scale – we can have situations where hundreds of millions of such tasks can be triggered in the span of a few hours which could result in millions of notifications being generated at constant rates nudging users about coupons nearing expiration.

To ensure coordination amongst containers, we would have to employ complicated strategies like partitioning of jobs and leader-election amongst task executor instances in all services handling these kinds of use cases. While possible, this would add a lot of complexity to the service containers themselves, making Garbage Collection, Thread pool, and auto-scaler tuning extremely difficult due to all containers doing mixed workloads, and adding significantly to the storage layer requirements for them.

As a design principle at PhonePe, we keep individual systems simple and build up complexity in layers, similar to building complex command chains by piping simple commands on Unix or GNU/Linux.

As this seemed like a fairly popular requirement across many systems we decided to build a centralized platform that can allow any clients to onboard and schedule jobs in the future without doing any kind of heavy lifting on their own. In this post, we will take a look at the internals of Clockwork - the system that powers job scheduling across various teams at PhonePe.

Scale

• Over 2 Billion callbacks are made daily as per the schedule defined.

• Capability to handle over 100,000 job schedules per second with a single-digit millisecond latency.

• No lag in the job execution at p99 which in the worst case can extend to 1 minute.

What is Clockwork?

Clockwork is a Distributed, Durable, and Fault-Tolerant Task Scheduler. Wow, that's a handful! Let's dissect it!

• Task Scheduler - In Linux, a job that needs execution at a specific point in time can be scheduled using the at command. In the case of recurring jobs, crontab can be used. Clockwork was designed based on that ideology, allowing clients to submit jobs that can be executed as per their schedule (once or repeated). Instead of executing arbitrary Java code, we limit it to only providing an HTTP callback to the provided URL endpoint at the specified time duration. Clients can schedule jobs that can be executed once, or at fixed intervals e.g. after every one day or daily at 5 PM.

• Distributed - To support high throughput of callbacks (100K RPS), we would need a service that can scale horizontally.

• Durable - Any submitted task is stored in a durable storage allowing Clockwork to recover from failures.

• Fault Tolerant - Any failure during job execution is handled gracefully per the client configuration. If a client wants the job to be retried, it is automatically retried upon failure as per its retry strategy.

Architecture - 1000-foot view

• Clients schedule Jobs.

• Jobs get executed.

• Clients receive callbacks.

• Profit 💰

Whenever a client schedules jobs, we store the job details in HBase and immediately send an acknowledgment back to the client.

Asynchronously, Clockwork keeps on performing HBase scans to find the list of eligible jobs that need execution. Once found, those jobs are immediately pushed to RabbitMQ (RMQ) to avoid blocking the scanner threads. As the callback is an HTTP callback to a URL endpoint, it can be time-consuming, hence it's important to decouple the job execution from the job extraction. Actual callbacks to the clients are performed in different threads after extracting the Job details from RMQ.

Worker threads subscribe to client-specific queues on RabbitMQ. In case of new messages, they are notified. Upon notification, a callback is made to the client as per the job details present in the message. If the callback is successful, an acknowledgment is sent to the RMQ which removes the message from the queue (acks the message). In case of a failure during callback (either because the downstream is down or an unexpected response code was received), retries are performed based on client-specific configuration.

Why HBase?

HBase is a key-value store structured as a sparse, distributed, persistent, multidimensional sorted map. This means that each cell is indexed by a RowKey, ColumnKey, and timestamp. Additionally, the rows are sorted by row keys. This allows us to efficiently query for rows by a specific key and run scans based on a start/stop key. We rely heavily on scans to find the candidate set of jobs whose scheduled execution time is less than or equal to the current time and callbacks are to be sent for.

Why RabbitMQ?

RabbitMQ is a messaging broker - an intermediary for messaging. It gives applications a common platform to send/receive messages and provides a durable place to store messages until consumed.

Architecture - Deep Dive

Clockwork service can be divided into 5 modules - each entrusted to perform a single responsibility.

Job Acceptor

Job Acceptor is a Client Facing Module. Its responsibility is to accept and validate the incoming requests from clients, persist the job details in HBase, and return an acknowledgment to the client. While persisting job details, a random Partition ID is assigned to the Job ID. We will cover the role of partitions in the subsequent sections.

Job Extractor

The Job Extractor’s responsibility is to find jobs that are eligible for execution. If the scheduled execution time of a job <= current time, a job becomes eligible. It finds eligible plans by running an HBase scan query between a time range. Once an eligible plan is found, the plans are pushed to RMQ one by one (without waiting for the job execution) to perform the next scan as soon as possible.

Leader Elector

At any point in time, there are multiple instances of Clockwork running. Each instance runs job extractors for all clients as all our containers are stateless. This poses a problem - if all of these extractors are trying to find eligible jobs for the same client at a given point in time, they all will get the same data, which will result in duplicate executions of the same job, something that cannot be allowed by any chance. The leader elector’s responsibility is to assign a leader amongst multiple clockwork instances for every client. The leader for a client assigns the partitions to the workers (extractors) running across different instances of clockwork.

• During the application startup, the application instance registers itself with Zookeeper with a unique worker ID.

• It then proceeds to check if there is a leader already elected for a client. If not, it tries to become a leader of a client.

• The client leader moves on to perform the partition assignment amongst existing clockwork instances (workers).

If you are still reading this article (kudos), you must have heard about the term Partition ID mentioned earlier. Why is it required? To support clients that need a lot of concurrent callbacks. Partitioning allows us to increase the number of concurrent scans that can be performed while ensuring a job only comes up in a single scan. If we have 64 partitions, we can perform 64 concurrent scans, allowing a higher rate of throughput.

But with great power comes great responsibility! We still have to ensure that no two instances scan the same partition. Otherwise, it will lead to double execution of the same job! This is where the partition assignment comes into the picture. The leader instance is responsible for assigning partitions to the workers - in a round-robin manner. This ensures fairness in partition distribution and ensures no two workers read the same partition.

RMQ Publisher

Once the list of eligible plans is fetched and some validations are performed, the messages are pushed to RMQ which acts as our message broker. While publishing the message we use Rate limiter.

Rate Limiter ensures that we don't publish more than what we can consume - otherwise, it can lead to the instability of the RMQ cluster - because of a huge backlog of messages. This is achieved by dynamically pausing scans if the queue size goes to a certain configurable threshold. Once the client can receive callbacks, and the queue size starts reducing, subsequent scans start pushing data into the queues.

We had to redesign this rate limiter to handle spiky traffic. Some of the clients schedule a lot of messages (>100k) that need execution at the same time. This leads to the fast producer - slow consumer problem, as the Job Extractor keeps finding the eligible jobs and enqueues to the RMQ queues so rapidly that the RMQ consumers are unable to catch up - leading to flow control and cluster instability across the RMQ cluster. To combat this, we used the Guava Rate Limiter to limit the publish rate to a limit that we know our 5-node RMQ cluster can handle (~100k consumer acknowledgement per second).

A lot of our workflows are time-sensitive, and delayed callbacks are sometimes not useful. Clockwork provides a way for clients to specify this time limit by setting a relevancy window. If the client is slow in accepting callbacks (slow consumer problem) or the scans were paused/slow during a time interval (slow publisher problem), expired callbacks will not be sent even when the system stabilizes. Besides this, we also support callback sidelining. This provides a way for clients to avoid getting overwhelmed by clockwork callbacks right after they recover.

RMQ Consumer

It listens to the incoming messages in the RMQ Queues and executes them by making an HTTP call to the specific URL endpoint. Any failure while making the call is handled by client-specific retry strategies.

Some clients don't want to retry and just want to drop failed messages, whereas some want to perform retries based on exponential backoff with random jitter. In case the retries are exhausted and the callback still fails, the message is pushed to a Dead-Letter-Queue. It's kept there until the messages are moved back to the main queue or the messages expire.

Conclusion

Clockwork's architecture is tailored and scaled to our specific needs and has enabled us to manage the enormous volume of tasks that PhonePe needs to handle efficiently. This system allows us to offer our customers seamless and uninterrupted service.

As engineers, we know that growth comes with challenges to infrastructure and system quality. At PhonePe, we are constantly seeking solutions to maintain that quality while managing costs and accommodating ever-increasing surges in traffic. Our goal is to share our learnings with the larger engineering community so that we can all learn how to address the challenges of growth and adapt our systems accordingly.

The link to the official blog article can be found here.

Please feel free to ask any questions you might have in the comments.

How does Linux handle directories and files?

What happens when you try to read a 10GB of file? How does Linux maintain the list of blocks to read from?

How is Linux able to list down thousands of files associated with a directory in seconds? Why is an empty directory always 4 KB?

In this blog, I will try to answer these questions in a simplified manner.

How is a file stored on the Disk?

Disk provides a block storage. It only deals with blocks and provides a facility to read and write blocks. If you ask it to read the contents of my cool movie.mkv it will get confused but if you ask it to return the content of the block number 20693, it will happily return you a series of 0's and 1's.

What is a block?

Block is the smallest unit of bytes that the disk can read/write in one go. If the block size is 4 KB, it can contain max 4 KB of data. If you modify a single bit of that block, the disk will need to update the entire block as it cannot modify anything smaller than the block size.

If the block size is too high, it can lead to wasted space because even if you want to write one byte, you will occupy one entire block. Too small a block size can also be problematic, as the data will now be spread across too many blocks, causing random seeks to be slow.

How to maintain a list of blocks associated with a file?

Enter FileSystem! A filesystem e.g. FAT, NTFS, ext4, and ZFS provides abstraction over the underlying disks - it maintains a list of blocks a file is associated with, so an end user can read a file seamlessly. How does it do that?

The simplest way would be to only assign a contiguous set of blocks to a file i.e. if you want to write a 4 MB of file, the filesystem will ensure that you only get 1024 contiguous blocks of 4 KB each. However, this approach isn't practical because of fragmentation. The files can get modified which can increase its length. In that case, we would have to defragment (compact) the entire disk, and then proceed with the update - which can be time-consuming.

The second approach would be to use a linked list. A block will link to the next block of the same file. This approach suffers from slow performance due to random seek. If we want to jump to the 10th block of a file, we would have to read the first 9 blocks, which can be slow.

The third approach is to maintain a file allocation table. This strategy was used in the FAT file system.

When you initialize a filesystem, it creates a fixed-size index based on the number of blocks. When you create a file, that spans across multiple blocks, it keeps on updating the index of the current block to the next block.

This looks very similar to the linked list approach but the caveat is that this index is kept in memory. If you have to jump to the 100th block, you still need to traverse 99 array indices, but since it's in memory, random accesses are much faster.

The fourth approach is maintaining a file indexing structure. If the file spans across 2000 blocks, the easiest way would be to keep track of all those 2000 blocks in a list-like data structure - but this would be very memory-expensive.

What if we could introduce some kind of tiered indexes? For small-sized files, we can maintain a 1:1 mapping of blocks but for bigger files, we can maintain a single level of indirection and for even larger files a double/triple layer of indirection can suffice.

In the diagram below, 10 blocks keeps a direct mapping of 10 data blocks i.e. for a file with size <= 40 KB (10 * 4 KB), we can perform random operation in one go.

For files larger than 40 KB, we have multiple levels of indirection. Assuming our block size is 4 KB, and a block number can be represented by an integer (32 bits i.e. 4 Bytes), a block can have 1024 such integers (4 KB / 4 Bytes). With a single level of indirection, we can keep track of 1024 other blocks = 1024 * 4 KB = 4096 KB = 4 MB.

As we add multiple levels of indirection, we can keep track of files with sizes of multiple GB's - all with a fixed set of memory. Isn't that cool !

Image Source

A file system maintains this information in Inode or Index Nodes. Every file has one unique Inode associated with it. Consider the below file of size 4.4 GB It's split across 9163680 such blocks and those blocks are mapped to the INode number 2098469.

P.S. Note that the file name is not associated with the Inode because we can have multiple names of the same file using hard links.

How to maintain directories?

Directories allow us to maintain files in a hierarchical structure. So far, we have understood how to represent files in a file system, so let's deep dive into how we can model directories too.

In Linux, everything is a file. So directories can also be thought of as a file. So directories contain essentially a list of file names present under it.

When you create a new/empty directory, it still takes 4 KB of space - which is equivalent to the block size. As and when the files keep on increasing, so does the size of the directory itself. (Note that when I talk about the size of the directory, it does not mean the size of the files contained inside it).

In the below screenshot, I have created 256 0-byte files in the directory, post which the size of the directory increases to 20480 bytes = 5 blocks. Why? To maintain the list of file names present under it.

In the original Unix implementation, 16 bytes were reserved for one directory entry, 14 bytes for the file name and 2 bytes for the Inode number, resulting in a filesystem that supports file names up to 2^14 (16,384). Now, this has changed and varies across file system implementations like ext4 and ZFS.

Image Source

How to trace pathnames?

With the internal representation of directories in place, let's evaluate the scenario when you perform ls -l /home/movies/action/Interstellar.mkv

You start your search from the root directory. Go through the directory entries of the root directory present at Inode 20030 and see if any file name matches home. If yes, go to the Inode of 56969. Recursively, repeat the search until you find your target file which is present at Inode 60025. From the Inode, the file system can easily find the list of blocks the file is present in, and you can view the contents of that file.

In the above example, we performed a linear search i.e. we searched the directories one by one. But what if, the directories contain millions of files? In that case, the search will be extremely slow. To optimize for large directories, filesystems maintain an external index (similar to the one used by databases) to optimize the search e.g. ext4 uses Hash Tree implementation to maintain a directory index.

Bonus Section

If you are with me so far, thank you. Hope you learned something new today. In this section, I will list down random learnings that I found during the research of this article.

How to output to another terminal window?

In Linux, everything is a file, even the disk, sockets and processes. When you run a terminal, the input and output to it are also modeled as a file. Open two terminal windows, find their device ID using tty and use echo command to output to a remote terminal window. So cool!

How to find the list of free blocks and free Inodes?

Whenever you want to write data to a file, you need an available block? How to quickly find an available block?

Enter the humble BitMap. Filesystems commonly maintain a bitmap to track free blocks. Bitmaps consume only 1 bit for 1 block and hence can be easily kept in memory to perform quick operations.

ZFS - the cool kid in town

I am not going to write in detail about the ZFS but it's the cool kid in town with a completely new architecture like inbuilt snapshots (the ability to version documents as and when they change), has an inbuilt cache layer (ARC which supposedly provides a better hit ratio than LRU cache - the default used in page cache), uses Copy on Write semantics and provides inbuilt checksums using Merkle Trees.

References

• https://pages.cs.wisc.edu/~solomon/cs537-old/last/filesys.html

• https://github.com/suvratapte/Maurice-Bach-Notes/blob/master/4-Internal-Representation-of-Files.md

• https://en.wikipedia.org/wiki/Hard_link

• https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Directory_Entries

• https://www.sans.org/blog/understanding-ext4-part-6-directories/

• https://teaching.idallen.com/cst8207/13w/notes/450_file_system.html#TOC

• https://metebalci.com/blog/a-minimum-complete-tutorial-of-linux-ext4-file-system/

• https://www3.nd.edu/~pbui/teaching/cse.30341.fa18/project06.html

• https://en.wikipedia.org/wiki/ZFS

Imagine, you have a pool of workers and a set of jobs that need execution. How are you going to model it?

class Job {
    List<Worker> workers = new ArrayList<>();

    public void executeJob() {
        workers.forEach(worker -> worker.execute(this))
    }
}

The simplest strategy is to pass a list of workers to the Job and then invoke them one by one during execution.

The downside of this approach is that it tightly couples two different concepts - job and its execution together in the same class. It's assigning multiple responsibilities to the same class which violates SRP (Single Responsibility Principle).

Observer Design Pattern can simplify this solution!

public record Job(String jobId) {
}

public interface Observer<T> {
    void update(T data);
}

public interface Subject<T> {
    boolean addObserver(Observer<T> observer);

    boolean removeObserver(Observer<T> observer);

    void notifyObservers(T data);
}

public class JobDispatcher implements Subject<Job> {
    Set<Observer<Job>> observers = new HashSet<>();

    @Override
    public boolean addObserver(Observer<Job> observer) {
        return observers.add(observer);
    }

    @Override
    public boolean removeObserver(Observer<Job> observer) {
        return observers.remove(observer);
    }

    @Override
    public void notifyObservers(Job job) {
        observers.forEach(observer -> observer.update(job));
    }
}

public class JobExecutor implements Observer<Job> {
    UUID uuid;

    public JobExecutor() {
        uuid = UUID.randomUUID();
    }

    @Override
    public void update(Job data) {
        System.out.println(uuid + " is executing jobId " + data.jobId());
    }
}

public class Main {
    public static void main(String[] args) {
        testObserver();
    }

    public static void testObserver() {
        JobExecutor executor1 = new JobExecutor();
        JobExecutor executor2 = new JobExecutor();
        JobDispatcher jobDispatcher = new JobDispatcher();
        jobDispatcher.addObserver(executor1);
        jobDispatcher.addObserver(executor2);
        jobDispatcher.notifyObservers(new Job("job1"));

        jobDispatcher.removeObserver(executor1);
        jobDispatcher.notifyObservers(new Job("job2"));
    }
}

// Output
3139088c-07eb-4f92-92da-0fb3af53b6be is executing jobId job1
fea0bf8a-6b2e-413f-9ae9-4a286b142798 is executing jobId job1
fea0bf8a-6b2e-413f-9ae9-4a286b142798 is executing jobId job2

We have created two new interfaces, Observer and the Subject. The sole responsibility of the Observer is to act when a subject is modified. The subject keeps track of the list of observers and decides how/when to invoke them when it changes.

What's the benefit?

• Adheres to SRP - Each class has one responsibility and is decoupled from another. The logic of execution can change without impacting the subject.

What's the drawback?

• This design pattern is very similar to the Publisher/Consumer problem and hence it suffers from all its issues as well. In a production ready code, we would have to be agnostic of various conditions like how to deal with slow observers, backpressure propagation and task backlog management.

In real-world applications, we frequently have to run a series of validations to ensure our model class is properly created before we persist it in our database. Consider the below Employee class

public record Employee(int employeeId, String firstName, String lastName, int salary, int managerId, int age) {
}

If we have to run validations to ensure whether the names are properly set, the age is valid or a valid employee ID has been assigned before we persist our POJO into a database, what's the simplest way?

Presenting the humble If/else!

  if (validEmployee.employeeId() != 0 
        && !validEmployee.firstName().isEmpty() 
        && !validEmployee.lastName().isEmpty() 
        && validEmployee.age() >= 18) {
      return true;
  }

What red flag do you see in the above code? Although it's simpler, it violates OCP (Open/Closed Principle). The code is not extensible and will require frequent modifications.

Chain of Responsibility Design Pattern can help tidy up the code.

public abstract class Validator {
    public Validator nextValidator;

    public Validator setNextValidator(Validator next) {
        this.nextValidator = next;
        return this;
    }

    public abstract boolean isValid(Employee employee);
}

public class EmployeeIdValidator extends Validator {
    @Override
    public boolean isValid(Employee employee) {
        System.out.println("Running Employee ID Validator");
        if (nextValidator == null) {
            // if there is no next validator in the chain
            return isIdValid(employee.employeeId());
        } else if (isIdValid(employee.employeeId())) {
            // delegate to the next validator
            return nextValidator.isValid(employee);
        } else {
            System.out.println("Employee ID is invalid");
            return false;
        }
    }

    private boolean isIdValid(int id) {
        return id != 0;
    }
}

public class NameValidator extends Validator {
    @Override
    public boolean isValid(Employee employee) {
        System.out.println("Running Name Validator");
        if (nextValidator == null) {
            return isNameValid(employee);
        } else if (isNameValid(employee)) {
            return nextValidator.isValid(employee);
        } else {
            System.out.println("Employee name is invalid");
            return false;
        }
    }

    private static boolean isNameValid(Employee employee) {
        return !employee.firstName().isBlank()
                && !employee.firstName().isEmpty()
                && !employee.lastName().isEmpty()
                && !employee.lastName().isBlank();
    }
}

public class AgeValidator extends Validator {
    @Override
    public boolean isValid(Employee employee) {
        System.out.println("Running Age Validator");
        if (nextValidator == null) {
            return isAgeValid(employee.age());
        } else if (isAgeValid(employee.age())) {
            return nextValidator.isValid(employee);
        } else {
            System.out.println("Age is not valid");
            return false;
        }
    }

    private boolean isAgeValid(int age) {
        return age >= 18 && age <= 70;
    }
}

public class Main {
    public static void main(String[] args) {
        testChainOfResponsibility();
    }

    public static void testChainOfResponsibility() {
        Employee validEmployee = new Employee(1, "Snehasish", "Roy", 100, 100, 20);
        // Chain the validators
        Validator validatorChain = new EmployeeIdValidator()
                .setNextValidator(new AgeValidator()
                        .setNextValidator(new NameValidator()));
        System.out.println(validatorChain.isValid(validEmployee));

        Employee invalidEmployee = new Employee(1, "Snehasish", "Roy", 100, 100, 10);
        System.out.println(validatorChain.isValid(invalidEmployee));
    }
}

// Output
Running Employee ID Validator
Running Age Validator
Running Name Validator
true

Running Employee ID Validator
Running Age Validator
Age is not valid
false

In the code above, we created dedicated validators for handling only one validation at a time. Each validator first performs local validations and then delegates to the next validator (if any).

What's the benefit?

• Follows OCP - Any modification/extension in the validation logic of a validator will require a change in only one class.

• Follows SRP (Single Responsibility Principle) - Each validator performs only one task.

What's the drawback?

• Validators must be chained correctly - if there are any cycles in the chain, then it can cause issues at runtime.

• The responsibility of initializing validators lies with the client. The client can either create a chain at the compile time or dynamically update the chain as per the business logic.

Functional programming heavily prefers Immutability i.e. objects should not be mutated. But what about the cases when we need to modify a specific field of an object? You need to create a copy of the entire object and update that specific field.

Okay, but how do I copy an existing object? Prototype design pattern to the rescue!

public class Car implements Cloneable {
    private final String name;

    private final List<Integer> mileage;

    public Car(String name, List<Integer> mileage) {
        this.name = name;
        this.mileage = mileage;
    }

    // Clone using Copy constructor
    public Car(Car existing) {
        this.name = existing.name;
        // deep copy
        this.mileage = List.copyOf(existing.getMileage());
    }

    // Clone using the default clone method.
    @Override
    public Car clone() {
        try {
            // by default provides shallow cloning i.e. only clones the references
            return (Car) super.clone();
        } catch (CloneNotSupportedException e) {
            throw new AssertionError();
        }
    }

    public String getName() {
        return name;
    }

    public List<Integer> getMileage() {
        return mileage;
    }
}

In the above code, we demonstrated two ways to create a clone of an Object in Java - using a custom copy constructor or by implementing the Cloneable interface.

Cloneable is a very special interface in Java - it's a marker interface - it does not have any method. When an Object implements Cloneable, JVM changes the protected() nature of the clone() method in Object class to public() - without which it throws a CloneNotSupportedException. By default, it provides a shallow clone i.e. only copies the reference.
Using Cloneable to create clones is highly discouraged because of its complicated nature.

Copy Constructors on the other hand are very intuitive and allow you to exactly control the behaviour. You can tweak the code to easily use shallow or deep copying as and when required.

public class Main {
    public static void main(String[] args) {
        testPrototype();
    }

    public static void testPrototype() {
        List<Integer> mileage = new ArrayList<>();
        mileage.add(10);
        mileage.add(20);
        Car originalCar = new Car("bentley", mileage);
        Car shallowClone = originalCar.clone();
        Car deepClone = new Car(originalCar);

        List<Integer> clonedMileage = shallowClone.getMileage();
        System.out.println(shallowClone.getName() + " " + clonedMileage);
        mileage.set(0, 50);
        // updating the mileage also updated the clonedMileage because they both 
        // point to the same object (shallow copy)
        System.out.println(clonedMileage.get(0).equals(50));

        // updating the mileage didn't affect the deep cloned object
        System.out.println(deepClone.getMileage().get(0).equals(10));
    }
}
// Output
bentley [10, 20]
true
true

Bonus Section

If you are using Lombok in your application to create boilerplate Java code, then you can use @With or @Builder(toBuilder = true) to create clones.

@With allows you to create a cloned object by updating only a specific field whereas toBuilder() allows you to override as many fields as possible. You can chain multiple with() to update multiple fields in succession but that would create a lot of garbage.

What's the benefit?

• Domain logic to clone the object does not spill out to the outer application.

What's the drawback?

• Circular dependency can become tricky to resolve. You would need to exclude a specific field to resolve the conflict.

• Need to be aware of the type of cloning being performed - shallow/deep.

References

• https://www.baeldung.com/lombok-builder

• https://projectlombok.org/features/With

If you want to dynamically make decisions in your class based on certain actions, then what’s the best way to achieve this?

public class OrderedList<T> {
    List<T> list;

    public OrderedList(List<T> list) {
        this.list = list;
    }

    public List<T> sort(String algo) {
        if ("bubble".equals(algo)) {
            return bubbleSort(list);
        } else if ("merge".equals(algo)) {
            return mergeSort(list);
        } else {
            throw new RuntimeException("Unsupported algo " + algo);
        }
    }
}

In the above code, if you need to change the sorting strategy based on the function argument, then the simplest way would be to use if/else and do a dispatch to the correct method body based on the argument.

Why would this be a problem?

Because, in the future, if you need to add a new strategy, you would have to update this class - which would violate the OCP (Open/Closed Principle) of the SOLID principle.

Solution?

public class OrderedList<T> {
    List<T> list;
    SortingStrategy<T> strategy;
    public OrderedList(List<T> list, SortingStrategy<T> strategy) {
        this.list = list;
        this.strategy = strategy;
    }

    // You can also update the code to take in Strategy at the time of sorting itself
    public List<T> sort() {
        return strategy.execute(list);
    }
}

public interface SortingStrategy<T> {
    List<T> execute(List<T> list);
}

public class MergeSort<T> implements SortingStrategy<T> {
    @Override
    public List<T> execute(List<T> list) {
        System.out.println("Performing merge sort");
        return list;
    }
}

public class BubbleSort<T> implements SortingStrategy<T> {
    @Override
    public List<T> execute(List<T> list) {
        System.out.println("Performing bubble sort");
        return list;
    }
}

public class Main {
    public static void main(String[] args) {
        testStrategy();
    }

    public static void testStrategy() {
        List<Integer> list = Arrays.asList(1, 2, 3, 4);
        BubbleSort<Integer> bubbleSort = new BubbleSort<>();
        MergeSort<Integer> mergeSort = new MergeSort<>();
        OrderedList<Integer> orderedList1 = new OrderedList<>(list, bubbleSort);
        orderedList1.sort();
        OrderedList<Integer> orderedList2 = new OrderedList<>(list, mergeSort);
        orderedList2.sort();
    }
}
// Output
Performing bubble sort
Performing merge sort

Using the strategy pattern, you can create new strategies and update the behavior of the class without modifying the class!

What's the benefit?

• Follows the OCP principle - class should be open for extension but closed for modification. You can change the behavior of the class by injecting a new strategy at runtime.

• Follows the SRP (Single Responsibility Principle) - class should have only one responsibility. Only one strategy is present in one class.

• Follows the Composition over Inheritance principle suggested by GoF (Gang Of Four).

What's the drawback?

• Creating new classes every time can create clutter if not done judiciously. Many languages allow you to create anonymous classes e.g. lambdas in Java - which can reduce the boilerplate code creation.

Consider the below Animal interface. If you want to add new methods to the interface, you have to update Cow and Dog classes to implement those methods.

public interface Animal {
}

public class Cow implements Animal {
}

public class Dog implements Animal {
}

But what if you want to add new functionalities across all the classes without worrying about changing them, then Visitor Design Pattern is perfect for you.

public interface Animal {
    <T> T accept(AnimalVisitor<T> visitor);
}

public class Cow implements Animal {
    @Override
    public <T> T accept(AnimalVisitor<T> visitor) {
        return visitor.visit(this);
    }
}

public class Dog implements Animal {
    @Override
    public <T> T accept(AnimalVisitor<T> visitor) {
        return visitor.visit(this);
    }
}

In the above scenario, let's say you want to add new functionality like Speak or NumberOfLegs, then you have to create an implementation of AnimalVisitor and implement your business logic directly in those classes.

public interface AnimalVisitor<T> {
    T visit(Cow cow);

    T visit(Dog dog);
}

public class LegsVisitor implements AnimalVisitor<Integer> {
    @Override
    public Integer visit(Cow cow) {
        return 4;
    }

    @Override
    public Integer visit(Dog dog) {
        return 4;
    }
}

public class SpeakVisitor implements AnimalVisitor<String> {
    @Override
    public String visit(Cow cow) {
        return "Moo";
    }

    @Override
    public String visit(Dog dog) {
        return "Bark";
    }
}

Driver Code

public class Main {
    public static void main(String[] args) {
        Animal cow = new Cow();
        Animal dog = new Dog();
        AnimalVisitor<Integer> legsVisitor = new LegsVisitor();
        AnimalVisitor<String> speakVisitor = new SpeakVisitor();

        System.out.println(cow.accept(legsVisitor));
        System.out.println(cow.accept(speakVisitor));
        System.out.println(dog.accept(legsVisitor));
        System.out.println(dog.accept(speakVisitor));
    }
}

Output:
4
Moo
4
Bark

What's the benefit?

• If you refer to the code once more, then you will notice that we added new functionality to Cow and Dog class without changing it -- this functionality is critical when working with client libraries.

• All the business logic is encapsulated in Visitor classes, which can help in segregating domain logic from model classes e.g. LegsVisitor contains all the logic for calculating the number of legs for all types of animals.

• Adherence to Open-Closed Principle i.e. class is open to extension but closed for modifications.

• Adherence to the Single Responsibility Principle i.e. class has only one responsibility.

What's the drawback?

• If you have a lot of sub-classes, then the Visitor implementation must handle them even if you want to add new functionality to only one of the classes.

• While adding a new class, you must update the existing visitor implementations to support the new class.

Welcome back to the third part of our tutorial series on building a distributed job scheduler! In our previous installment, we deep-dived into our storage system by designing a durable storage system to store job details effectively. Now it's time to model repeated jobs and handle job executions.

Modeling Job Execution

Recapping the discussion from the first part of the series

Our platform should support three types of jobs:

• Once - These jobs need to be scheduled only once at a specified date and time, such as scheduling a job for August 1, 2023, at 23:00.

• Repeated - Repeated jobs occur within a defined date range, with a specified time interval between each occurrence. For example, scheduling a job to run every 30 minutes between August 1 and August 31, 2023.

• Recurring - Recurring jobs are scheduled for specific dates and times, e.g., on August 1, 2023, at 16:00, and on August 5, 2023, at 12:30.

This indicates that there is a one-to-many relationship between jobs and the way they are executed.

Introducing the Plan Class

Since a job can be executed multiple times, we need to introduce another model to track the execution history of a job.

Whenever a job is scheduled, we will insert a Plan corresponding to the Job in the Plan Table. We will monitor the Plan table continuously to fetch eligible Plans. Once a Plan is fetched, we will fetch the corresponding Job details too, and generate the next Plan e.g. in case of a recurring Job scheduled on August 1 and on August 5 - Initially, the Plan with the expected execution on August 1 will be created. Once that plan is executed, we will generate the next plan with expected execution on August 5 will be created. This is known as lazy evaluation and will prevent unnecessary insertions into the Plans table in case of a repeated job that runs for years.

public class Plan implements Serializable {
    String planId;
    String jobId;
    LocalDateTime expectedExecutionTime;
}

Updating the JobDAO Class

JobDAO class needs to be updated to account for the creation and storage of newly created Plan table. getJobDetails() has also been updated to return all the PlanIDs mapped with the provided JobId.

public void registerJob(Job job) throws IOException {
    // Create and store the Job in HBase
    byte[] row = Bytes.toBytes(job.getId());
    Put put = new Put(row);
    put.addColumn(columnFamily.getBytes(), data.getBytes(), SerializationUtils.serialize(job));
    hBaseManager.put(table, put);

    // Create and store the Plan in HBase
    Plan plan = planDAO.storePlan(job);
    PlanIDs planIDs = PlanIDs.builder()
            .plans(List.of(plan.getPlanId()))
            .build();

    Put jobPlanPut = new Put(row);
    jobPlanPut.addColumn(columnFamily.getBytes(), plans.getBytes(), SerializationUtils.serialize(planIDs));
    hBaseManager.put(table, jobPlanPut);
    log.info("Generated JobId {} and PlanID {}", job.getId(), plan.getPlanId());
}

@VisibleForTesting
public JobDetails getJobDetails(String id) throws IOException {
    Result result = hBaseManager.get(table, id);

    byte[] jobValue = result.getValue(columnFamily.getBytes(), data.getBytes());
    Job job = (Job) SerializationUtils.deserialize(jobValue);

    byte[] plansMapping = result.getValue(columnFamily.getBytes(), plans.getBytes());
    PlanIDs planIDs = (PlanIDs) SerializationUtils.deserialize(plansMapping);

    return JobDetails.builder()
            .job(job)
            .planIDs(planIDs)
            .build();
}

Introducing the PlanDAO Class

This will be similar to JobDAO and will be responsible for dealing with the CRUD operations related to Plan entity.

public class PlanDAO {
    HBaseManager hBaseManager;
    String columnFamily = "cf";
    String data = "data";
    String tableName = "planDetails";
    Table table;

    public PlanDAO() throws IOException {
        hBaseManager = new HBaseManager();
        hBaseManager.ensureTable(tableName, columnFamily);
        table = hBaseManager.getTable(tableName);
    }

    public Plan storePlan(Job job) throws IOException {
        // Use of VISITOR pattern to extend functionality of an existing class
        Plan plan = job.accept(new PlanGenerator());

        byte[] row = Bytes.toBytes(plan.getPlanId());
        Put value = new Put(row);
        value.addColumn(columnFamily.getBytes(), data.getBytes(), SerializationUtils.serialize(plan));
        hBaseManager.put(table, value);
        return plan;
    }

    public Plan getPlanDetails(String planId) throws IOException {
        Result result = hBaseManager.get(table, planId);
        byte[] value = result.getValue(columnFamily.getBytes(), data.getBytes());
        return (Plan) SerializationUtils.deserialize(value);
    }
}

Visitor Pattern in Real Life

If you are with me so far, you must have noticed a sneaky new class called PlanGenerator.

What does it do? Based on the type of Job, it returns a Plan.

Why do we need a visitor? If we had not gone with this approach, then either we would have to use if/else logic to generate a Plan based on the instance of Job object or use Strategy pattern to generate a Plan.

public interface JobVisitor<T> {
    T visit(ExactlyOnceJob job);

    T visit(RecurringJob job);

    T visit(RepeatedJob job);
}

public class PlanGenerator implements JobVisitor<Plan> {
    @Override
    public Plan visit(ExactlyOnceJob job) {
        return Plan.builder()
                .jobId(job.getId())
                .planId(getRandomID())
                .expectedExecutionTime(job.getDateTime())
                .build();
    }

    @Override
    public Plan visit(RecurringJob job) {
        return Plan.builder()
                .jobId(job.getId())
                .planId(getRandomID())
                .expectedExecutionTime(job.getDateTimes().first())
                .build();
    }

    @Override
    public Plan visit(RepeatedJob job) {
        return Plan.builder()
                .jobId(job.getId())
                .planId(getRandomID())
                .expectedExecutionTime(getNextExecutionTime(job.getStartTime(), job.getRepeatIntervalTimeUnit(), job.getRepeatInterval()))
                .build();
    }

    private String getRandomID() {
        return UUID.randomUUID().toString();
    }

    private LocalDateTime getNextExecutionTime(LocalDateTime start, TemporalUnit repeatIntervalTimeUnit, long repeatInterval) {
        return start.plus(repeatInterval, repeatIntervalTimeUnit);
    }
}

Tests

Finally, the tests to verify the integration.

public class JobDAOTest {

    JobDAO jobDAO;
    PlanDAO planDAO;

    @Before
    public void setUp() throws IOException {
        jobDAO = new JobDAO();
        planDAO = new PlanDAO();
    }

    @Test
    public void testRegisterJob() throws IOException {
        JobDAO jobDAO = new JobDAO();
        String jobId = UUID.randomUUID().toString();
        ExactlyOnceJob exactlyOnceJob = ExactlyOnceJob.builder()
                .id(jobId)
                .callbackUrl("http://localhost:8080/test")
                .successStatusCode(500)
                .build();

        Assertions.assertDoesNotThrow(() -> jobDAO.registerJob(exactlyOnceJob));
        JobDetails jobDetails = jobDAO.getJobDetails(jobId);
        Assertions.assertTrue(exactlyOnceJob.equals(jobDetails.getJob()));
        PlanIDs planIDs = jobDetails.getPlanIDs();
        Assertions.assertNotNull(planIDs);

        String planId = planIDs.getPlans().get(0);
        Plan plan = planDAO.getPlanDetails(planId);
        Assertions.assertEquals(planId, plan.getPlanId());
        Assertions.assertEquals(jobId, plan.getJobId());
    }
}

Appendix

Project Structure

Conclusion

Congratulations! In this third part of our tutorial series, we've made significant progress. We modeled multiple executions of the same job, implemented the necessary code, used Strategy pattern in real life and validated it through test cases. But the journey doesn't end here. In the next installment, which we'll cover in part 4, we'll delve into an even more complicated part - identifying which Plans to execute and executing them. Do take a pause and think about the various challenges that can come in fetching Plans from the Plan table. Stay tuned for more exciting insights!

References

• Part 2 of the series

• Part 1 of the series

Welcome back to the second part of our tutorial series on building a distributed job scheduler! In our previous installment, we laid the foundation by defining the functional and non-functional requirements of our job scheduler. Now, it's time to dive into the heart of our system by designing a durable storage system to store job details effectively. If you're a software engineer eager to learn new technologies, this tutorial is tailored just for you.

Modeling Job Class

Since we have already figured out what are the various Job types and the ways to configure callbacks, our actual job has become quite easier.

public abstract class Job implements Serializable {
    String id;
    // The actual HTTP url where callback will be made.
    String callbackUrl;
    int successStatusCode;
    long relevancyWindow;
    // Defines the maximum window for callback execution
    TimeUnit relevancyWindowTimeUnit;
}

public class ExactlyOnceJob extends Job {
    LocalDateTime dateTime;
}

public class RecurringJob extends Job {
    List<LocalDateTime> dateTimes;
}

public class RepeatedJob extends Job {
    LocalDateTime startTime;
    LocalDateTime endTime;
    TimeUnit repeatIntervalTimeUnit;
    long repeatInterval;
}

SQL vs NoSQL

Before figuring out the database, let's figure out the query patterns.

• Store job details - We need high write throughput to store structured data. Additionally, we must be prepared for possible schema changes in the future.

• Get job details provided an ID - High read throughput to get details of a job based on a key.

• No transaction guarantees are required.

• No range scans are required.

Considering these requirements, we can choose a NoSQL database like Cassandra or HBase. For this tutorial, we'll leverage Apache HBase due to its capabilities.

Hello, HBase!

If you are new to the world of HBase, I would recommend you to read this crisp and excellent article which would give you a fair idea of the HBase data model.

Installing HBase is a 5-minute affair and can be completed relatively easily. Just go through this link.

Now it's time to write some boilerplate utility code to interact with our newly created HBase Server.

public class HBaseManager {

    private Admin admin;
    private Connection connection;

    public HBaseManager() throws IOException {
        Configuration config = HBaseConfiguration.create();
        String path = Objects.requireNonNull(this.getClass().getClassLoader().getResource("hbase-site.xml"))
                .getPath();
        config.addResource(new Path(path));
        HBaseAdmin.available(config);
        connection = ConnectionFactory.createConnection(config);
        admin = connection.getAdmin();
    }

    public boolean tableExists(String name) throws IOException {
        TableName table = TableName.valueOf(name);
        return admin.tableExists(table);
    }

    public void createTable(String name, String columnFamily) throws IOException {
        if (!tableExists(name)) {
            TableName table = TableName.valueOf(name);
            HTableDescriptor descriptor = new HTableDescriptor(table);
            descriptor.addFamily(new HColumnDescriptor(columnFamily));
            admin.createTable(descriptor);
        }
    }

    public Table getTable(String name) throws IOException {
        TableName tableName = TableName.valueOf(name);
        return connection.getTable(tableName);
    }

    public void put(Table table, Put value) throws IOException {
        table.put(value);
    }

    public Result get(Table table, String id) throws IOException {
        Get key = new Get(Bytes.toBytes(id));
        return table.get(key);
    }
}

Now that our utility code is in place, we can proceed to create a Data Access Object (DAO) layer responsible for storing and retrieving job details.

 public class JobDAO {
    HBaseManager hBaseManager;
    String columnFamily = "cf";
    String data = "data";
    String tableName = "jobDetails";
    Table table;

    public JobDAO() throws IOException {
        hBaseManager = new HBaseManager();
        hBaseManager.createTable(tableName, columnFamily);
        table = hBaseManager.getTable(tableName);
    }

    public void registerJob(Job job) throws IOException {
        byte[] row = Bytes.toBytes(job.getId());
        Put put = new Put(row);
        put.addColumn(columnFamily.getBytes(), data.getBytes(), SerializationUtils.serialize(job));
        hBaseManager.put(table, put);
    }

    public Job getJobDetails(String id) throws IOException {
        Result result = hBaseManager.get(table, id);
        byte[] value = result.getValue(columnFamily.getBytes(), data.getBytes());
        Job job = (Job) SerializationUtils.deserialize(value);
        return job;
    }
}

To ensure the functionality of our system, we'll rely on JUnit tests to validate our code. This step is crucial to confirm that our storage system works as expected.

public class JobDAOTest {
    @Test
    public void testRegisterJob() throws IOException {
        JobDAO jobDAO = new JobDAO();
        String id = UUID.randomUUID().toString();
        ExactlyOnceJob exactlyOnceJob = ExactlyOnceJob.builder()
                .id(id)
                .callbackUrl("http://localhost:8080/test")
                .successStatusCode(500)
                .build();
        Assertions.assertDoesNotThrow(() -> jobDAO.registerJob(exactlyOnceJob));
        ExactlyOnceJob job = (ExactlyOnceJob) jobDAO.getJobDetails(id);
        Assertions.assertTrue(exactlyOnceJob.equals(job));
    }
}

Appendix

Project Structure

Maven pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>
    <groupId>com.scheduler</groupId>
    <artifactId>scheduler</artifactId>
    <version>1</version>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>8</source>
                    <target>8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
    <dependencies>
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-client</artifactId>
            <version>2.5.3</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase</artifactId>
            <version>2.5.3</version>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>1.18.28</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>3.12.0</version>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>5.2.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-runner</artifactId>
            <version>1.2.0</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
</project>

Conclusion

Congratulations! In this second part of our tutorial series, we've made significant progress. We've chosen a suitable database (HBase), implemented the necessary code, and validated it through test cases. But the journey doesn't end here. In the next installment, which we'll cover in part 3, we'll delve into modeling repeated jobs. Do take a pause and think about why they need to be modeled separately. Stay tuned for more exciting insights!

References

• Part 1 of the series

• Part 3 of the series

• Understanding HBase and Bigtable

• How to Install Apache HBase on Ubuntu

In this multi-part series, we're rolling up our sleeves to build a robust distributed job scheduler from scratch. Get ready to dive into the world of distributed systems!

Understanding the requirements

Before we jump into the technical details, let's establish a clear understanding of what our distributed job scheduler needs to achieve.

Job Types

Our platform should support three types of jobs:

• Once - These jobs need to be scheduled only once at a specified date and time, such as scheduling a job for August 1, 2023, at 23:00.

• Repeated - Repeated jobs occur within a defined date range, with a specified time interval between each occurrence. For example, scheduling a job to run every 30 minutes between August 1 and August 31, 2023.

• Recurring - Recurring jobs are scheduled for specific dates and times, e.g., on August 1, 2023, at 16:00, and on August 5, 2023, at 12:30.

How to configure callbacks?

To offer flexibility, our system must allow clients to configure various aspects of their callbacks:

• Retry strategies - Defines what happens if a callback fails. Should it be retried, and if so, what should the retry strategy entail?

• Auth token - Provide an authentication token for client-side verification during callbacks.

• Callback Path/URL - The actual HTTP URL where callback will be made.

• Headers to pass - Any custom application headers to pass during the callback.

• Success status codes - How to interpret whether the callback succeeded? Simply relying on 200 won't suffice for all the client use cases.

• Relevancy window - Defines the maximum window for callback execution e.g. expected time of callback is 13:00 but that job somehow got picked up at 13:30, Is that job still valid? This can be configured by the client by providing the relevancy window. If the relevancy window <= 30 minutes, the callback can be performed, otherwise, it can be skipped.

With these functional requirements in mind, let's move on to considering some non-functional aspects that will shape our system.

Durability

If a client has received a successful acknowledgment of a job being accepted from our platform, job details must be persisted in durable storage.

Scalability

Design various components of the overall system keeping scalability in mind. Keep the components loosely coupled so that one can be scaled independently of the other.

Callback Guarantees

In a distributed system, it's difficult to make guarantees - especially exactly once, so let's go with it i.e. ensure at least one callback of the scheduled job to our client. Clients might receive multiple callbacks but it's up to them to identify and decide whether or not to process those duplicate requests.

Conclusion

Congratulations on making it this far! In this first part of our tutorial series on building a distributed job scheduler, we've outlined the essential functional and non-functional requirements. Let's take a pause and think about how are we going to implement a job scheduler based on the above requirements. I will not directly go into drawing boxes and assigning responsibilities to those boxes - instead, we will first figure out what kind of work we actually have to do and then we will decide whether or not we need a component to handle these kinds of tasks.

In the next installment, we will start by constructing a durable storage system that can persist job details efficiently and allow for quick lookups based on job IDs.

Link to part 2: https://snehasishroy.com/building-a-distributed-job-scheduler-from-scratch-part-2

Hidden within the canvas of reality, treasures often lie in plain sight, awaiting the patient heart and the perceptive eye to unveil their concealed brilliance.

Amid this whirlwind modern world, mastering EPUB-to-PDF conversion via Calibre can feel like trying to win a game of hide-and-seek against a mischievous squirrel. Sure, Calibre’s got all the bells and whistles, but sometimes we just want to hit that “PDF” button and call it a day, right?

Fret not! I’m here to guide you through the virtual maze and turn those electronic pages into glorious PDFs without losing your sanity. Grab your magnifying glass and let’s dive into these settings that will make your PDFs shine.

Where to start your journey?

Download and Setup Calibre. All the screenshots provided are based on v6.24.

Right-click any book in your library and choose “Convert Individually” present under “Convert Books”.

Now select PDF as the output format in the top right-hand corner.

Enable Text Justification

Ever seen text that’s wandering all over the page like a kid who’s had too much sugar? We’re going to whip it into shape.

Head to the Look and Feel section, then navigate to Text, and there you’ll find the Text Justification option. Choose “Justify Text” like a pro, and watch those lines of text straighten up and fly right. It’s like a font intervention.

Conversion -> Look and Feel

Enable Image Resizing

Images that refuse to fit in their designated boxes are like trying to fit an elephant into a phone booth. To conquer this puzzle, venture into the Search and Replace section. There, you’ll encounter the enigmatic “Search Regular Expression.” It’s like a decoder ring for resizing images dynamically.

Add in the below secret code and watch those images slide into place like puzzle pieces.

\ width=\"(.*?)\" height=\"(.*?)\"

Conversion -> Search and Replace

Format the PDF

Now, the grand PDF Output stage awaits your artistic touch.

Select your paper size, much like choosing the canvas for your masterpiece. I went with the trusty A4. Do not select the “Use the paper size set in output profile”

If you’re into numbers, add page numbers at the bottom, like breadcrumbs for digital travelers.

Choose your Serif, Sans Family and Monospace fonts (like picking the right outfit for your document’s fancy party). I chose the good old Calibri and Courier family.

Adjust font sizes, so your text doesn’t look like ants marching across the page.

Oh, and be sure to let the document’s original margins have their moment in the spotlight. In case you want to override the margins, unselect the “Use Page Margins from the document being converted” and provide in your overrides. Here is a handy link that can convert pts into inches. (72 pts is 1 inch)

Conversion -> PDF Output

Bonus Tip: Update Default Preferences

Who has time to repeat the same process for every single document? Not you, that’s for sure! Head over to Preferences like a boss. Tweak your conversion settings here once, and they’ll swoop in like a superhero whenever you’re converting more files.

Preferences Section

Preferences -> Conversion -> Common Options

Preferences -> Conversion -> Output Options

Wrapping Up

And there you have it, dear PDF pioneers! Calibre might have thrown some curveballs your way, but armed with these settings and a good sense of humor, you’re now equipped to navigate its twists and turns like a pro.

So go forth, and create PDFs that’ll have your readers thinking, “This is the work of a true PDF Picasso.”

In case you find something new and want to share, please comment.

Thank you for reading :)

References

1. https://calibre-ebook.com/

2. https://maxrohde.com/2015/01/27/rendering-beautiful-pdf-documents-with-calibre

3. https://github.com/lorenzodifuccia/safaribooks/issues/22

4. https://www.thecalculatorsite.com/conversions/length/points-to-inches.php

There is something mystical in physical books that an ebook can never replace.

Picture this: you’re browsing through an e-commerce store in India, eying those coveted O’Reilly paperback books, your excitement building. But hold that thought! There’s a hidden caveat that could dent your wallet and leave you unimpressed. The culprit? The official seller - notorious for peddling lackluster books at exorbitant prices. Don’t be fooled by their fancy facade — your hard-earned money deserves better!

Now, before you swear off your book-buying dreams, I'm here to reveal a game-changing solution that'll leave you with premium reads and extra cash in your pocket. It's time to roll up your sleeves and embark on a journey to book nirvana.

Step 1: Create an account on Oreilly

First things first, we’re sidestepping the Publisher and heading straight to the source. Forge yourself a dummy account on Oreilly.com, and voilà! You’ve unlocked 10 days of trial account bliss. Run into expiration trouble? A swift signup shuffle will sort you right out.

Step 2: Clone SafariBooks Git Repository

Clone this amazing git repo https://github.com/lorenzodifuccia/safaribooks and follow the below instructions

$ git clone https://github.com/lorenzodifuccia/safaribooks.git
Cloning into 'safaribooks'...

$ cd safaribooks/
$ pip3 install -r requirements.txt

$ python3 safaribooks.py --cred "account_mail@mail.com:password01" XXXXXXXXXXXXX

Provide the username and password of your account on oreilly.com followed by the Book ID.
The ID is the digits that you find in the URL of the book description page:
https://www.safaribooksonline.com/library/view/book-name/XXXXXXXXXXXXX/
Like: https://www.safaribooksonline.com/library/view/test-driven-development-with/9781491958698/

At the end of this step, you will have received a genuine and up-to-date epub copy of the book you are trying to read.

P.S: In case you are facing login issues in the script, have a look at this PR.

Step 3: Download and Install Calibre

Next stop, Calibre — a world of wonder masked by its humble UI. Drop your newly acquired ePub gem into Calibre’s treasure trove.

Step 4: Convert EPUB into PDF

Epub format internally uses HTML/CSS to represent the text, structure and format of the content document. It’s extremely customizable as it’s pretty flexible. Don’t like the font, you can change it. Don’t like the spacing, you can change it. Want to change the background, believe it or not, you can change it too.

PDFs on the other hand are static. It does not provide customizations but is extremely good for printing because it doesn’t change. You can print a pdf from any computer and they would come the same because pdf is sort of a digital paper. What you see is what you get.

If you need to print a book, you will most likely need its pdf version. So we would need to convert the epub into pdf. You need not download additional software and can use Calibre itself. I have written another article which provides a step-by-step way to do the same.

Step 5: Printing as a Service

The crescendo approaches, as your customized masterpiece yearns for the tactile embrace of paper. Enter online book printing services like Printster, your partner-in-printing. Get ready to call the shots — spiral binding for frugality? Check. Soft-cover lightness? Check. Hard-cover extravagance? Double check. With options galore and competitive pricing, your dream book metamorphoses into reality.

Look at that subtle off-white coloring. The tasteful thickness of it.

In my quest for bookish bliss, I ventured with 85 GSM Bond paper, a robust color cover, monochrome elegance, dual-sided splendor, and the pièce de résistance — hardcover binding. The result? An awe-inspiring tome at a fraction of Amazon’s asking price.

Cover image at its glory

Crisp off-white 85 GSM Paper at its beauty

Dear O’Reilly, if you’re listening, heed the plea: a new publisher is overdue.

Armed with this revelation, I hope you’re ready to embark on your budget-friendly book-buying escapades. Say goodbye to those inflated prices and lackluster editions — a brave new world of high-quality, wallet-happy reads awaits. Happy reading, and may your literary journey be ever-illuminating!

References

1. https://www.oreilly.com/

2. https://github.com/lorenzodifuccia/safaribooks

3. https://calibre-ebook.com/download

4. https://snehasishroy.medium.com/unleash-your-inner-pdf-whisperer-with-calibre-conquering-the-epub-to-pdf-expedition-da2a36b83e42?source=friends_link&sk=6b4fdca67d92ccd78bb9906c00bd5354

5. https://printster.in/

Disclaimer

The information provided in this article is for educational and informational purposes only. The author and publisher of this article are not responsible for any actions taken based on the information presented herein. The content of this article does not constitute professional advice, and readers are advised to consult with appropriate professionals before making any decisions or taking any actions.

The references to specific companies, products, or services in this article are purely for illustrative purposes and do not constitute endorsements or recommendations. The author and publisher do not have any affiliation with the companies mentioned, and the information provided is based on publicly available sources as of the time of writing.

Readers are encouraged to independently verify the accuracy and relevance of the information provided in this article. Any reliance on the information contained in this article is at the reader’s own risk. The author and publisher disclaim any liability for any loss, damage, or inconvenience caused by reliance on the information provided herein.