The title has a funny typo.

What is the purpose of the persistent cache ? How is it different from the base chronik storage ?

In D18437#420303, @Fabien wrote:

The title has a funny typo.

What is the purpose of the persistent cache ? How is it different from the base chronik storage ?

There will be a pure in-memory mode. However, between fully storage like chronik and pure in-memory, there should also be an "on-demand persistence" option. Both in the browser and in Node.js environments, this kind of flexible persistence is often needed.

In D18437#420334, @alitayin wrote:

In D18437#420303, @Fabien wrote:

The title has a funny typo.

What is the purpose of the persistent cache ? How is it different from the base chronik storage ?

There will be a pure in-memory mode. However, between fully storage like chronik and pure in-memory, there should also be an "on-demand persistence" option. Both in the browser and in Node.js environments, this kind of flexible persistence is often needed.

"Because it's often needed" doesn't answer the question though. Why is it often needed ?

In D18437#420336, @Fabien wrote:

In D18437#420334, @alitayin wrote:

In D18437#420303, @Fabien wrote:

The title has a funny typo.

What is the purpose of the persistent cache ? How is it different from the base chronik storage ?

There will be a pure in-memory mode. However, between fully storage like chronik and pure in-memory, there should also be an "on-demand persistence" option. Both in the browser and in Node.js environments, this kind of flexible persistence is often needed.

"Because it's often needed" doesn't answer the question though. Why is it often needed ?

Many use cases require repeated access to specific data. For example, the address used by paybutton, or the token used by agora. Storing all of these in memory is not a good option. Selective persistence can better complement memory usage.

In D18437#420340, @alitayin wrote:

In D18437#420336, @Fabien wrote:

In D18437#420334, @alitayin wrote:

In D18437#420303, @Fabien wrote:

The title has a funny typo.

What is the purpose of the persistent cache ? How is it different from the base chronik storage ?

There will be a pure in-memory mode. However, between fully storage like chronik and pure in-memory, there should also be an "on-demand persistence" option. Both in the browser and in Node.js environments, this kind of flexible persistence is often needed.

"Because it's often needed" doesn't answer the question though. Why is it often needed ?

Many use cases require repeated access to specific data. For example, the address used by paybutton, or the token used by agora. Storing all of these in memory is not a good option. Selective persistence can better complement memory usage.

Storing all of these in memory is not a good option

Why ? Isn't it exactly the point of a cache ?
The goal of a cache is to avoid a costly IO operation by replacing it with a less expensive one, typically store in memory to avoid a disk lookup which is orders of magnitude slower. Another example, browser cache is about replacing networking IO with a local disk lookup.
What is your persistent cache doing ? What is the rationale behind it ?

In D18437#420348, @Fabien wrote:

In D18437#420340, @alitayin wrote:

In D18437#420336, @Fabien wrote:

In D18437#420334, @alitayin wrote:

In D18437#420303, @Fabien wrote:

The title has a funny typo.

What is the purpose of the persistent cache ? How is it different from the base chronik storage ?

There will be a pure in-memory mode. However, between fully storage like chronik and pure in-memory, there should also be an "on-demand persistence" option. Both in the browser and in Node.js environments, this kind of flexible persistence is often needed.

"Because it's often needed" doesn't answer the question though. Why is it often needed ?

Many use cases require repeated access to specific data. For example, the address used by paybutton, or the token used by agora. Storing all of these in memory is not a good option. Selective persistence can better complement memory usage.

Storing all of these in memory is not a good option

Why ? Isn't it exactly the point of a cache ?
The goal of a cache is to avoid a costly IO operation by replacing it with a less expensive one, typically store in memory to avoid a disk lookup which is orders of magnitude slower. Another example, browser cache is about replacing networking IO with a local disk lookup.
What is your persistent cache doing ? What is the rationale behind it ?

Yes, the goal of caching is indeed to replace expensive operations with cheaper ones, which is exactly my design purpose.

Let me explain the design evolution process:
Initial Design: I originally designed two approaches for different environments:
Browser Environment: Persistent cache + Chronik API
Node.js Environment: Memory cache + Chronik API (hot data/cold data model)
Problem Encountered: pure memory caching faces size bottlenecks. For tokens or addresses with high transaction volumes, cached data easily reaches hundreds of MBs. While data loss after restarts isn't a primary concern (since it's infrequent), memory capacity limitations became a practical obstacle.

Therefore, I introduced the concept of "warm data" as an extension solution,, which is LevelBD

So let me sum up to check I understand: you have persistent storage on front-end and both memory and persistent on backend. So the persistent storage on backend is only useful if the chronik instance is a remote to save the network io?

In D18437#420414, @Fabien wrote:

So let me sum up to check I understand: you have persistent storage on front-end and both memory and persistent on backend. So the persistent storage on backend is only useful if the chronik instance is a remote to save the network io?

Yes, when designing it in February of this year, I was considering "the situation where there's no locally running chronik node", which is why I thought of using levelDB to persist some data. However, I think the overall structure indeed has many shortcomings, so I want to have a thorough discussion on how to improve its design before starting to improve it. the purpose is not just to save resources, but rather to provide a solution for "faster access to desired data" instead from "network"

Data Access Solution Design
├── Scenario 1: Pure Frontend Solution
│ ├── Feature: No local chronik node
│ ├── Storage: levelDB persistence
│ └── Goal: Fast client-side data access
│
├── Scenario 2: Backend without Chronik Solution
│ ├── Feature: Backend service, no chronik dependency
│ ├── Storage Options:
│ │ ├── Persistent storage
│ │ └── Pure in-memory storage
│ └── Goal: Flexible data management
│
└── Scenario 3: Backend with Chronik Solution

├── Feature: Backend service with existing chronik node
├── Storage: No need for levelDB persistence
└── Goal: Direct utilization of chronik data source

In D18437#420414, @Fabien wrote:

So let me sum up to check I understand: you have persistent storage on front-end and both memory and persistent on backend. So the persistent storage on backend is only useful if the chronik instance is a remote to save the network io?

You're right, I was explaining the original design approach above. It does have some issues, which is why it hasn't been integrated yet.

The original solution didn't have decoupling problems, but it was more oriented towards:

How to fetch all data in one go
How to cache this data, then write a complex mechanism to ensure data consistency between levelDB and chronik
The cache is only considered "complete" and usable after all transactions for an address or token are fully retrieved
Designed specifically for a few tokens/addresses with large transaction volumes
Assumed most users are chronik API users who wouldn't deploy chronik themselves
Assumed that when a user accesses address A, all records for address A would be asynchronously fetched to form a robust, complete dataset, which inherently consumes extra resources and only generates "benefits" when address A is requested frequently
These design tendencies made it a "prewarming" tool with limited use cases.

So the new approach is:

Everything is processed in memory with no DB. Browser-side data storage uses localStorage
For small amounts of data, there's no need for persistence
For large amounts of data, users should deploy chronik rather than creating duplicate mirrors through persistence
No extra or duplicate data requests, zero redundancy, fetch on-demand, lightweight state, maintaining transparent proxy behavior - for users, it's simply a faster chronik client.
The original design is indeed very useful for "specific scenarios", and I might rename that original design, but chronik-cache needs to switch to the new approach.

I will design a new solution and then present it to you offline.

In D18437#420414, @Fabien wrote:

So let me sum up to check I understand: you have persistent storage on front-end and both memory and persistent on backend. So the persistent storage on backend is only useful if the chronik instance is a remote to save the network io?

My previous designs were not atomic enough; I turned them into the tools I wanted according to my preferences. This is a problem with my design, I will improve. It's just like my ecash-quicksend; the difference between it and ecash-wallet is also here: it's like an upper layer on top of ecash-wallet, you can make a tx easily, but that is all, It can't fulfill any "requirements" other than this. this is how i understand "atomic"

In D18437#420437, @alitayin wrote:

In D18437#420414, @Fabien wrote:

So let me sum up to check I understand: you have persistent storage on front-end and both memory and persistent on backend. So the persistent storage on backend is only useful if the chronik instance is a remote to save the network io?

You're right, I was explaining the original design approach above. It does have some issues, which is why it hasn't been integrated yet.

The original solution didn't have decoupling problems, but it was more oriented towards:

How to fetch all data in one go
How to cache this data, then write a complex mechanism to ensure data consistency between levelDB and chronik
The cache is only considered "complete" and usable after all transactions for an address or token are fully retrieved
Designed specifically for a few tokens/addresses with large transaction volumes
Assumed most users are chronik API users who wouldn't deploy chronik themselves
Assumed that when a user accesses address A, all records for address A would be asynchronously fetched to form a robust, complete dataset, which inherently consumes extra resources and only generates "benefits" when address A is requested frequently
These design tendencies made it a "prewarming" tool with limited use cases.

So the new approach is:

Everything is processed in memory with no DB. Browser-side data storage uses localStorage
For small amounts of data, there's no need for persistence
For large amounts of data, users should deploy chronik rather than creating duplicate mirrors through persistence
No extra or duplicate data requests, zero redundancy, fetch on-demand, lightweight state, maintaining transparent proxy behavior - for users, it's simply a faster chronik client.
The original design is indeed very useful for "specific scenarios", and I might rename that original design, but chronik-cache needs to switch to the new approach.

I will design a new solution and then present it to you offline.

The new architecture will shift to an "on-demand caching + transparent proxy" model:
Maintain metadata (such as total transaction count numTxs) for each address/token, used as a "fingerprint" for cache validity.
Calculate the precise position of data in the cache based on pageOffset and pageSize.
Listen for new transactions via WebSocket. When numTxs changes, the cache becomes invalid.

Remove LevelDB and use in-memory caching; browser clients can optionally use localStorage.
Workflow example:

User requests page 2 of data for address A (pageOffset=1, pageSize=200)
System quickly checks and finds that the address has a total of 1337 transactions
Calculate cache position: transactions 200-399, and check whether this range is already cached
If cache miss, fetch via chronik API and place precisely in the corresponding position
WebSocket detects 3 new transactions and updates numTxs to 1340
Next time the same request is made, it detects the change in numTxs and fills in the missing 3 transactions
Cache size can be set, and the lifecycle is determined according to request volume
For the client, use localStorage. The idea is similar, also using LRU for automatic eviction.

Subscriptions also have their own management process to avoid subscribing to too much content.

In short, the core is to ensure that the obtained data maps to the correct cache slot, that its order and offset are accurate, and to achieve this with minimal resources :-) can also add UTXO, block(maybe)

In D18437#420534, @alitayin wrote:

In D18437#420437, @alitayin wrote:

In D18437#420414, @Fabien wrote:

So let me sum up to check I understand: you have persistent storage on front-end and both memory and persistent on backend. So the persistent storage on backend is only useful if the chronik instance is a remote to save the network io?

You're right, I was explaining the original design approach above. It does have some issues, which is why it hasn't been integrated yet.

The original solution didn't have decoupling problems, but it was more oriented towards:

How to fetch all data in one go
How to cache this data, then write a complex mechanism to ensure data consistency between levelDB and chronik
The cache is only considered "complete" and usable after all transactions for an address or token are fully retrieved
Designed specifically for a few tokens/addresses with large transaction volumes
Assumed most users are chronik API users who wouldn't deploy chronik themselves
Assumed that when a user accesses address A, all records for address A would be asynchronously fetched to form a robust, complete dataset, which inherently consumes extra resources and only generates "benefits" when address A is requested frequently
These design tendencies made it a "prewarming" tool with limited use cases.

So the new approach is:

Everything is processed in memory with no DB. Browser-side data storage uses localStorage
For small amounts of data, there's no need for persistence
For large amounts of data, users should deploy chronik rather than creating duplicate mirrors through persistence
No extra or duplicate data requests, zero redundancy, fetch on-demand, lightweight state, maintaining transparent proxy behavior - for users, it's simply a faster chronik client.
The original design is indeed very useful for "specific scenarios", and I might rename that original design, but chronik-cache needs to switch to the new approach.

I will design a new solution and then present it to you offline.

The new architecture will shift to an "on-demand caching + transparent proxy" model:
Maintain metadata (such as total transaction count numTxs) for each address/token, used as a "fingerprint" for cache validity.
Calculate the precise position of data in the cache based on pageOffset and pageSize.
Listen for new transactions via WebSocket. When numTxs changes, the cache becomes invalid.

Remove LevelDB and use in-memory caching; browser clients can optionally use localStorage.
Workflow example:

User requests page 2 of data for address A (pageOffset=1, pageSize=200)
System quickly checks and finds that the address has a total of 1337 transactions
Calculate cache position: transactions 200-399, and check whether this range is already cached
If cache miss, fetch via chronik API and place precisely in the corresponding position
WebSocket detects 3 new transactions and updates numTxs to 1340
Next time the same request is made, it detects the change in numTxs and fills in the missing 3 transactions
Cache size can be set, and the lifecycle is determined according to request volume
For the client, use localStorage. The idea is similar, also using LRU for automatic eviction.

Subscriptions also have their own management process to avoid subscribing to too much content.

In short, the core is to ensure that the obtained data maps to the correct cache slot, that its order and offset are accurate, and to achieve this with minimal resources :-) can also add UTXO, block(maybe)

You don't need metadata, and you should start with memory only. Offloading to disk via the LRU algo can be done later if needed which I doubt will ever be. Also local storage is browser side so it's never gonna be useful and you can avoid it.

The transparent proxy (aka the cache extends chronik-client and uses the same api) is good and indicates it should be part of the client.

You want to keep it as simple and useful as possible: I would go with a read through cache design that is marked as dirty by the ws notifications, + LRU for limiting the memory usage.