The CoE Numerai Newsletter - Edition #17
Oct "Fireside Edition" Q4 2023
WE. ARE. SO. BACK.
After a year long hiatus, the newsletter is returning, with StudyM8 filling in for Aventurine. If this is your first time reading, make sure to subscribe below to receive your monthly newsletter.
Project Maple Leaf: Toronto Meetup
The latest instalment of the “Around the world with Numeratis” series was held in Toronto last month. The event started with a brief networking breakfast, then moved into introductory panels by Aventurine. After lunch, it continued with “Language Models: From Words to Numbers” by SurajP and “Using the Shiny Numerati Dashboard to Find Your Best Models” by Ai-Joe.
The content and recordings from the meetup can be found on GitHub.
If you’d like to attend a CoE event near you, a survey for upcoming events is now open here!
Benchmark models are live!
by Keno
The predictions and details of various standard Numerai benchmark models are now available to tournament participants. The performance of these models can be found under the NUMERAI BENCHMARK MODELS user.
The benefits of benchmark models were outlined in a forum post by Mike P:
Why?
New User Acceleration
Numerai has a steep learning curve. After you make it through the tutorial notebooks, you are left with several datasets, many targets, and many modeling options. There are an unlimited number of experiments you’ll want to run as you begin your journey to the top of the leaderboard. With benchmark models, you can immediately see how well different combinations of data and targets do. I think you’ll find that exploring these models and their predictions and subsequent performance will inspire even more ideas for new models you can build yourself.
Better Stake Allocation
If you’re a returning user and you’re a few updates behind, you can see at a glance if your model is still competitive, or if you’d be better off staking on one of the newer benchmark models until you have time to catch back up.
A Meta Model of Meta Models
Some users may not have the resources to train large competitive cutting-edge models themselves. However, by just downloading targets, the Meta Model predictions, and Benchmark Model predictions, it may still be possible to recognize that the Meta Model is underweight some types of models, or you might be able to find that certain targets ensemble especially well together, or you might have a strong belief that one target will outperform into the future. You can explore all of these possibilities yourself and even submit and stake on these ensembles with minimal resource requirements.
If you’d like to learn more about how to leverage the benchmark models, please see the documentation.
Numerai Q4 2023 Fireside Chat Highlights
Model Uploads
Off to a great start with ~2200 staked models in the first 6 months and roughly 15K NMR staked through model uploads (1.5% of all stakes). If you haven’t tried them yet, check them out here.
Rain Data Release
First dataset with synthetic ML generated features that users can't create themselves due to data obfuscation. These features are made by looking at the time-series of features with filters to maximize performance.
ML used to engineer features that have desirable characteristics (E.g. Lower churn, lower correlation to existing features, boost performance on multiple targets).
Training of these newly generated features stops in 2014.
Rain showing better draw-down vs previous datasets.
New data releases are "superset" of previous data releases, E.g. V2 data lives within Rain data.
666 new features, and smallest in memory constraints (GBs) as INT formatting is used.
Smaller memory size allows for a better example model that’s trained on the medium-size feature set.
Please see the forum post by Mike P for more information on the Rain data (Rain Data Release).
Signals Integration to Meta Model
Signals models are now a part of the meta model.
Multiple targets have been added to Signals, similar to the classic tournament.
The Great Burn of 2023
Current total NMR staked at 75% of all-time high due to burns.
Many users are not beating example preds.
Upcoming scoring changes to disincentivize stakers that perform worse than example predictions, as they reduce the Payout Factor for better performers.
Q&A
Q (Anonymous): Wen stake management?
A:
TLDR: Not happening any time soon.
Rapidly changing stakes between models to chase performance adds churn to the meta model, and is not a strategy Numerai is looking to encourage.
Q(Correlator): Can more effort be put into giving guidance for improving TC. Users lack visibility of the entire trading setup and the team is in the place to do so.
A: Being good at leader board metrics is a good indicator for TC performance. Chasing TC isn't worth the time investment.
Q(PsyRex): Any post-mortem results on the nose dive in funds performance? Steps taken/planned to avoid that happening again? guidance/tips for participants on how to help?
A: This draw-down was very sharp and took a while to recover in performance. Example preds burning during the same period pushed the team to analyze targets and create new targets that would’ve been immune to this draw-down.
CoE Transaction Report
Oct 4th 2023 -15 NMR Numerbay
Oct 5th 2023 -143 NMR Toronto Meetup
All transactions from the CoE wallet can be viewed at the following link using SAFE.
Disclaimer: This is not an official Numerai newsletter. It is sponsored by the Numerai CoE, a decentralized autonomous organization. Every effort is made to provide accurate and complete information but there are no claims, promises or guarantees about the accuracy of the contents.