I have lots of questions, let's start with a couple of the main ones:

• What is the purpose of the cron and the reconciliation thing, why is there several data gathering methods (ws, cron, reconciliation) ?
• How would you plug a new chart/data collection to this ?

In D18307#417987, @Fabien wrote:

I have lots of questions, let's start with a couple of the main ones:

• What is the purpose of the cron and the reconciliation thing, why is there several data gathering methods (ws, cron, reconciliation) ?

It was added when I was doing initial indexing to ensure the initial indexing was not skipping rows. The initial indexing involved so many API calls and iterative run-ups against memory limits that I had to do this a few times, and it was worth just having it automated. At this point in the codebase, I do not think it is necessary for the initial indexing (if the server crashes, it corrects on restart) -- but since the primary way of continuing to index is websockets, there should be some kind of system that handles a missed websocket msg.

In practice I have not seen any missed websocket msgs. But the check is not expensive and does make everything more robust.

The issue causing row skipping crashes during initial indexing -- making API calls based on number of blocks, not on number of txs in those blocks, and then crashing when X number of blocks has way more txs than expected -- is solved; we now query for next blocks based on a capped number of txs.

• How would you plug a new chart/data collection to this ?

1. Add a new column to the db
2. Update the schema in the codebase to match
3. Develop a system to index empty columns that are not price in the background (i.e. websockets add new rows with all data, a routine is indexing empty columns with chronik calls until we have data for all blocks)
4. On indexing complete, modify the front end to chart the new data

Step 3 is doing a lot of lifting here. But, since all of the data comes from chronik calls, I don't think it would be too difficult. I think type of approach would be easier / less downtime than a full reindex of all data, tho that would also work. I can imagine some things may require a full reindex, but in that event we could just plug a new instance to a new db, wait for it to index, then switch the charts front end to the new db with the new columns.

Recommended path forward here

• Test the charts app at https://charts-e-cash.vercel.app/ and recommend any launch-critical changes
• Review metachronik as an analytics (non-financial, non-critical, kind of experimental) app.

metachronik is mostly AI-generated, but the indexing components are tested (i.e. how we go from typed chonik responses to the data we want to store). It is not a speed-optimized indexer. I plan to add many new and potentially time-consuming columns to this database. I will modify the codebase so that new columns can be added "in the background," i.e. without impacting existing columns in the database.

Leveraging AI will be able dramatically increase the pace of iterations. I think that this type of system is appropriate for this type of product, which otherwise never would have been built.

I'm also not really sure what the alternative would be like. So much about how the indexer is designed depends on what we want to index. We do not know exactly what we want to index yet, this is expected to be dynamic. It seems pretty wild to have a "fluid" indexer. But for an experimental / analytics-focused meta indexer...why not? The database is not expected to be fluid. As our schema matures, we will have a better time optimizing for indexing itself.

• Keep the current deployment infra

• Charts is currently deployed on vercel free tier. we could bump it if / when we have a traffic problem. Vercel is capable of handling custom domains, so we could launch this at charts.e.cash, or any other domain
• metachronik is currently deployed on https://railway.com/ hobby tier ($5/month). It's hard to say what kind of server demands we will have with this app. Easy to scale and iterate rapidly on something like railway.
• the database is at neon.tech ($20/month). We could host our own db for this, but the neon front end is pretty useful, especially when dealing with expected troubleshooting, bugs, other unforeseen issues involved in working out new columns. it's easy to link a nextjs app or a railway indexer to a postgresql database on neon.

vercel and railway both support CI deployments, so we could still automate deployments through teamcity.

There are still some axis issues in the charts. Example: open the block size chart and unzoom: the axis go very large and you can't zoom back to an appropriated scale without reseting

In D18307#418445, @Klakurka wrote:

Different issues but there's a problem with the 'Price Data' page 'All Time' (showing pricing pre-2020) and 'BCH' (showing wrong pricing) filters.

should be patched now

In D18307#418465, @Fabien wrote:

There are still some axis issues in the charts. Example: open the block size chart and unzoom: the axis go very large and you can't zoom back to an appropriated scale without reseting

patched

in this case, specific issue related to this chart -- the database has this value in bytes, but we are transforming it to KB for the chart. the zoom function was not doing the same transform.

cron only checks empty blocks. Perhaps a completion mechanism for "newly added parsed data" can be added, which would make it easier to add new parsed data in the future. Neon responds very quickly when selecting data ranges in sqp, which is better than a self-hosted database. As for the frontend, an auto-resize function or button can be added to the sidebar, as well as adapting it for mobile devices (with smaller font sizes).

In D18307#418493, @alitayin wrote:

cron only checks empty blocks.

Why ?

imo the indexing function is overengineered already and looking for further optimizations is not the right move (arguably we should take stuff out, but what's in there was only added bc it proved necessary or useful during the initial indexing process).

• On startup we make sure there are no missed blocks
• We have daily reconciliations to make sure we have no missed blocks
• Every time a new block comes in from the ws, we make sure it's the one we expected; if not we fill in missing blocks
• Every 6 hrs we confirm we are on the chaintip, if not, gap detection, fill us in

It's been empirically tested as this stuff was added following issues experienced with in the initial indexing, including trying to do this all on a free low-memory VPS.

arguably we should get rid of it and make this an easier review. The unique thing I am asking for in this diff review: treat the indexing functionality itself as malleable and an early-stage iteration. This is a bit un-natural, since for chronik itself and for the node software indexing is an incredibly critical function that must be optimized and fault tolerant in all kinds of ways. In this case though, the indexing is not the main point of this app at all. The performance of the indexer is not as important as getting the desired data eventually. We do not expect users or even (most) devs to want to run this, it is designed for devs and hobby analysts.

I plan to add more columns to this database, which will likely drastically overhaul some of the indexing and require its own routines. This type of approach would be stupid if not for the ability to do it with some speed thanks to AI support and the ability to test the indexing functionality itself on individual or batched blocks, which come in strongly typed chronik shapes.

This app is not a production critical indexer. The product is the database itself, and the quality of the database will be confirmed by the quality of the indexing tests. Because chronik blocks and txs are strongly typed, we can have good confidence in the quality of the data. The indexing functionality itself depends a lot on what we want to index -- which we do not exactly know quite yet -- so the indexing functionality is expected to continue changing.

We are optimizing for quantity and flexibility of data, not indexing performance.

In D18307#419388, @Klakurka wrote:

On mobile (portrait), the buttons for XEC, BCH, etc are missing.

IMO not important since we mostly care about XEC here anyway but just FYI.

could make it more clear -- this area is swipe scrollable on mobile

In D18307#419395, @bytesofman wrote:

In D18307#419388, @Klakurka wrote:

On mobile (portrait), the buttons for XEC, BCH, etc are missing.

IMO not important since we mostly care about XEC here anyway but just FYI.

could make it more clear -- this area is swipe scrollable on mobile

In some browsers the scrollbar is obvious at page load but in some it's not.