Why We Publish the Numbers Instead of Just Saying It’s Lightweight
There is nothing to verify in a claim that something is lightweight. mote commits to eight numbers that speak for it, and publishes the methods, conditions, and raw data alongside them.
Markdown editors are often introduced like this.
Fast. Lightweight. Smooth.
But when numbers omit their conditions, they are no different from adjectives. When idle CPU was measured, and for how long. Where the measurement of startup time began and ended. Because numbers measured under different conditions cannot be compared.
So we settled on eight performance metrics that mote publishes. For each metric, we documented the measurement method and conditions, exclusions, comparison rules, caveats, and budget. We also linked each one to its measurement script.
The documentation, results, and code all use the same names. That makes it possible to immediately trace where each number came from.
Numbers Should Come with Conditions
mote follows four principles.
For every measurement, we store the value and unit, statistic, sample count, and raw samples together. Results without units are rejected at the validation stage.
We do not report values we have not measured. If something cannot be measured, we leave it blank and explain why. We do not fill it with an estimate.
We make comparisons only on the same machine and within the same session. We do not place numbers from different days or different devices side by side.
We publish the measurement conditions along with the numbers. In particular, we always record the refresh rate and CPU governor. If these conditions differ, latency and startup time cannot be compared properly.
We did not invent new measurement methods.
For the latency from a keystroke to a change on the screen, we use the same method employed by Pavel Fatin’s Typometer. We generate keystrokes from outside the app and watch until the pixels change. Like VS Code, we divide startup time into stages and distinguish cold starts from warm starts.
The principle that latency should be measured end to end, rather than from within the framework, comes from Dan Luu’s article.
We Compared Them Ourselves Under the Same Conditions
We did not cite performance numbers for competing apps from elsewhere. We measured them ourselves on the same laptop.
We opened the same 100 KB document and ran each app seven times with the same script. All of them used an inline editing view, with the pointer placed in the same position.
The total memory usage of their processes was as follows.
- mote: 78 MB
- Typora: 453 MB
- Obsidian: 361 MB
- MarkText: 416 MB
CPU usage while typing, measured against a single core, was 10.9%, 103%, 152%, and 136%, respectively. Their process counts were 1, 8, 7, and 6, respectively.
mote used 4.6 to 5.8 times less memory. It used 9.4 to 14 times less CPU while typing.
We calculated memory usage using PSS rather than RSS. RSS can count memory shared by multiple processes more than once. PSS divides shared memory according to the number of processes. It is also the method used by system monitors.
We Kept the Unfavorable Numbers Too
When the apps were merely left open, CPU usage was 0.0% for mote, 0.1% for Typora, 0.8% for Obsidian, and 0.6% for MarkText.
Typora also used almost no CPU while idle. Therefore, we cannot present “doing nothing when left open” as an advantage over Typora.
The web shell used 3.6%, making it worse than Typora.
Eliminating recurring timers inside the app is still important. But a principle that must be upheld and a competitive advantage are two different things.
We did not compare idle frames with competing apps. mote’s target is zero frames over ten seconds, but the other apps do not have frame counters that can be checked in the same way.
So we left the fields for competing apps blank. A blank is not zero.
If It Cannot Be Compared, We Do Not Compare It
We also check whether the original remains unchanged when a file is opened and then saved without modification.
mote tests 652 CommonMark examples. We also open and resave every document in the repository, then verify that it is identical byte for byte. If even one differs, we treat it as a data-loss bug, not a performance issue.
We did not report results for competing apps. That is because it was difficult to automate the save process under the same conditions.
We also check the state of the machine before measuring. We do not begin measurements while a compilation is running. We block the browser only when it is actually using a significant amount of CPU. The load average, CPU idle percentage, decision criteria, and values at the time of measurement are all retained in the results.
These rules came from experiencing failure.
When we measured startup time with a CPU-intensive browser running, it appeared that performance had degraded. After alternating measurements of the previous commit and the current commit under the same load, we found that the cause was machine load, not a code regression.
Since then, we have not judged startup time based on a single absolute value.
If It Exceeds the Budget, It Cannot Be Merged
Performance budgets are not merely goals written in documentation. They are requirements that must be met for the build to pass.
The criteria are divided into hard and report.
hard applies to items whose numbers have been established in decision documents. If even one exceeds its threshold, the check fails. There is no separate exception approval.
report applies to items for which evidence is still insufficient or that are heavily affected by platform limitations. Their values are recorded, but they do not block the build.
If you block builds with numbers that lack sufficient justification, people begin to avoid measurement itself. That is why we use only well-founded criteria as gates.
We keep the measurement scripts, raw CSV files, and result JSON files together in the repository. Running the same scripts under the same conditions reproduces the tables. We are also preparing to release the measurement tools and test documents in a public benchmark repository.
Publishing only the good numbers is not transparency.
Showing what was measured and how, under what conditions, and what could not be measured as well. That is how mote publishes its numbers.