The CoE Numerai Newsletter- Edition #4

November 1st "Fireside Edition" Q4 2021

by Numerai


Numerai Hedge Fund Performance Is Here!

On Oct 6th 2021 an announcement was made by the Numerai team that they have been quietly beating the most prestigious hedge funds in the industry.

Today, we are is excited to share the performance of our hedge fund for the first time with the data scientist community who built it.

A medium article written by Richard Craib was released entitled, “Numerai Outperforms Market Neutral Hedge Funds by 29%, Raises Up To $150m.”

The announcement and medium article created much excitement from the community and beyond.

Since strategy launch in September 2019, Numerai is beating Aurum’s Quant Equity Market Neutral Index by 29.14% and AQR’s Market Neutral Fund by 34.41%.

What’s Numerai’s AUM?

Numerai recently raised money from a new investor. They are Numerai’s first big institutional investor, a pension fund. They have a deal with Numerai where they gave them $20m but can invest up to $130m more later. Subsequent investments after the first $20m are at lower fees so they are incentivized to use their $130m in capacity rights to re-invest as new investors come on board.

Numerai’s current AUM is currently at about $43m right now so they are still very small for a hedge fund, but, Numerai’s fund trades with 5.5x leverage so almost one quarter of a billion dollars is controlled by Numerai’s data scientists right now.

Learn more or contact Numerai @ numerai.fund


Q3 End-of-Quarter Updates

Compute 0.3.1

Numerai added a new cron trigger functionality, meaning you can schedule your data pipelines or models to run in the cloud whenever you want. It also includes several bug fixes and infrastructure updates. Be sure to follow the upgrade guide to get these updates.

Signals Diagnostics

Numerai has migrated Signals to their new diagnostics architecture, meaning you can now find the new Diagnostics Tool at signals.numer.ai This contains all the Signals diagnostics you’re familiar with and is available 24/7 just like the Numerai Tournament. This is meant to replace the diagnostics you receive after submitting. The old method of receiving diagnostics will be discontinued in the coming weeks.

TB 200, Autocorr, Adjusted Sharpe, and APY

Numerai is introducing new diagnostics for both Tournament and Signals.

Top/Bottom 200 (TB 200) is a copy of all diagnostics on the top 200 stocks and the bottom 200 stocks you submit predictions for; the ones that represent your most extreme predictions.

Along with this new set of diagnostics Numerai will also be including Autocorrelation and an Adjusted Sharpe Ratio. This 1-era-lagged Autocorrelation (Autocorr) is useful in determining how repetitive your scores are over time - a high Autocorr (near 1) means that good scores this round show a higher probability of good scores next round, and vice versa. The Adjusted Sharpe Ratio accounts for skewness and kurtosis of your scores, meaning that you can get a better idea of the actual risk/reward ratio.

Finally, Numerai will be adding Annual Percentage Yield (APY) to the Classic Tournament diagnostics, to provide an idea of the potential profitability of your models. This will bring Tournament Diagnostics into parity with Signals Diagnostics.


End of October 2021 MORE Updates

New Feature Metadata

As of round 287, Numerai is making public the long awaited Feature Metadata file. This is downloadable each round through the dataset API and the dataset zip as “features.json”. Inside the file are two sections: “feature_stats” and “feature_sets”.

The 3 feature sets in the file are lists of feature names:

  • Legacy: 304 of the original 310 features that were carried over to the new dataset. You can use this set to achieve nearly the same model as the legacy data.

  • Small: 38 features that were found to be the most important based on Shapley values calculated by Michael Oliver in this forum post. You can achieve relatively high performance using these features without many compute resources. Check out our example model 28 to see how to use this!

  • Medium: 420 features that can capture a large amount of information that exists in our dataset. This is not a strict superset of the “small” dataset and does not use Shapley values to pick these.

Here’s an explanation of each statistic:

  • Legacy Uniqueness: how unique this feature is compared to the legacy dataset. This is 1 - max(corr(feature, legacy_features)), so a legacy_uniquness of 0 means that feature is perfectly correlated with at least one feature in the legacy dataset.

  • Spearman Correlation w/ Target Nomi (20-day): these are aggregated statistics over the per-era correlation between this feature and target_nomi_20. Mean and Sharpe (mean divided by standard deviation) are given as well as the following statistics:

    • Reversals: how many times the feature changes between correlated and uncorrelated with the target. A low value implies consistency of correlation.

    • Autocorr: Similar to autocorr from diagnostics, this is the correlation of lagged per-era corr values with themselves. If this is high, the feature is more likely to be consistently correlated (or uncorrelated) with the target.

    • Average Run Length (ARL): The average number of contiguous eras that this feature stays highly correlated with the feature.

Numerai-CLI 0.3.2

The newest version of the numerai-cli includes small updates to the way we build Dockerfiles allowing you to be more modular with your code and share local packages between nodes. It also provides support for Cron-based Nodes when running “numerai node test”.

Before this update, the CLI only gave Docker access to whatever folder you configured for a given node. Now, it gives Docker access to whatever parent directory you run “numerai node deploy” in. This is especially useful when copying code from a sibling directory; for an example of how to accomplish this, check the wiki

Deprecating Signals Submission Diagnostics

Until last quarter, the only way to get diagnostics for your signal was to submit it and wait over 10 minutes to see the results. Now that we have a new Diagnostics Tool (which runs in 2 minutes or less), we no longer need to run the expensive and redundant diagnostics when you submit.

Starting October 25, 2021, we will no longer be providing diagnostics when you submit a signal to Numerai, instead you can use the always-available diagnostics tool.

Coming This Quarter

For Q4 2021, Numerai wants to continue unification of Signals and Tournament by starting to score Tournament submissions against a 20D2L target (20 days long w/ a 2-day lag). This target is already included in the new data, Numerai is just switching their scoring to use it. This change will likely take place sometime in November.

We are also preparing more data. You all experienced the amazing Supermassive Data Drop last Quarter, but we aren’t done yet. Numerai Data V3.2 is coming December 25, 2021; prepare yourselves and your computers.


Memes of the Month

by wacax

by oftfrfbf


Numerbay Updates

Oct 4th: The following includes changes from the stake_mode branch and master branch which have been merged today. (Stake mode features are disabled until release, the eventual roll-out will be a turnkey operation.)

  • Completed basic submission scheduling and a test submission

  • Added basic product sales stats to the product page

  • Added email notification for: [Buyers: New Artifact Available]

  • Improved artifact management UI

  • Fixed model metrics display issues on the catalog page

  • Other issue fixes and minor improvements

  • Work plan is in post #2

  • Next week to focus on staking limit and robustness

Oct 11th: The following includes changes from the stake_mode branch and master branch which have been merged today. (Stake mode features are disabled until release, the eventual roll-out will be a turnkey operation.)

  • Completed stake limit and automated submission

  • Added minimal off-platform listing support for non-tournament categories (e.g. OnlyFams)

  • Introduced a new daily scores update mechanism. Scores should update within 1 minute after they become available

  • Added email notifications for: [Sellers: New Artifact Added, Failed Artifact Upload]

  • Added backend tests to maintain ~70% of coverage

  • Improved listing and checkout UI

  • Other issue fixes and minor improvements

  • Work plan is in post #2

  • Next week to rollout stake mode and start working on buyer/seller reputation systems

Oct 18th: The following includes changes from the reputation branch and master branch which have been merged today.

  • Completed stake mode rollout

  • Added artifact upload validation API

  • Added product readiness indicator

  • Added product review system

  • Added site-wide statistics

  • Fixed concurrent upload and scores update

  • Other issue fixes and minor improvements

  • Work plan is in post #2

  • Next week to start working on subscription sales and more flexible product pricing

Oct 25th: The following includes changes from the subscription branch which is not live yet.

  • Completed UI for multiple pricing options

  • Added API for managing pricing options

  • Added fallback to Numerai emails

  • Other issue fixes and minor improvements

  • Work plan is in post #2

  • Next week to work on backend for multiple pricing, code refactoring, order processing and management, API tests, integration testing and preparing for rollout

Oct 31st: The following includes changes from the subscription branch, mobile branch and wishlist branch.

  • Completed multi-option pricng rollout

  • Completed mobile UI optimization

  • Added related products section

  • Added wishlist

  • Other issue fixes and minor improvements

  • Work plan is in post #2

  • There is no more major feature to work on currently. Next week will mainly be routine maintenance and minor improvements


NumerBay Stake Mode Beta

Stake mode is now live for beta!

This is the final milestone for core features as laid out in the proposal. However, there will be other feature releases and enhancements from time to time.

What is Stake Mode?
Previously the on-platform sale on NumerBay requires distribution of predictions files to buyers and provides no means of automation for submissions. When a product is listed in stake modes, NumerBay submits for the buyers without distribution of raw files. Sellers can also choose to impose a Stake Limit for their customers.

Listing Modes

  1. Distribute File: Buyers can download files. Sellers can either upload file artifact or add external URL. Buyers can optionally choose to desginate a model slot for submission during checkout. The file to submit will be the latest CSV artifact file uploaded by the seller

  2. Stake Only: Buyers cannot download files. Sellers can only upload file as product artifact. Buyers must designate a model slot for submission during checkout.

  3. Stake Only with Limit: Same as above but with a stake limit for buyers. Target Stake Value is calculated as TARGET_STAKE = CURRENT_STAKED_AMOUNT + PENDING_STAKE_CHANGE. When TARGET_STAKE > STAKE_LIMIT, the pending stake change will be set to keep the target stake value below the stake limit.

(Optional) Artifact Validation Endpoint for API Users
A new artifact upload validation endpoint 1 is now available. API users can call this endpoint after uploading artifact to NumerBay to get immediate confirmation of upload success/failure, instead of waiting for 10 minutes. This makes the API workflow similar to Numerai’s (request upload URL → upload → validate). Buyers who have emails will receive notifactions for the new upload. Submissions will happen for confirmed orders immediately.

Please refer to the example notebook for details: https://github.com/councilofelders/numerbay/blob/master/NumerBay%20Example.ipynb


[New Feature] NumerBay Multi-option Pricing

NumerBay (https://numerbay.ai/) now supports multi-option pricing.

DISCLAIMER: NumerBay is in beta and may have unexpected issues. This is NOT an official Numerai project. Neither Numerai nor NumerBay will be liable for any loss.

What is Multi-option Pricing?
Previously listings on NumerBay can only be sold one round at a time at a fixed price. With the new pricing scheme, sellers can offer buyers more flexible options such as bundling sales for multiple rounds (prepaid subscriptions), making volume discounts, setting different prices for different modes, etc.

Please note the following:

  1. The “Price” field for each option refers to the total for that option, not the equivalent unit price per round. Total price for any option still needs to be above 1 NMR. Duplicated price is not allowed.

  2. Mixed on/off-platform listing is allowed. However, the first option is the default for display to buyers.

  3. Please remember to save the pricing option first before saving the listing form.

  4. NumerBay does not yet send out weekly reminders to upload artifacts. Sellers need to remember to do so if they have active subscriptions

  5. [Number of Rounds for Order] = [Pricing Option Bundled Quantity] x [Order Quantity] (E.g. a buyer who bought 3 of “2 x file @ 1.5000 NMR” option needs to pay 3 x 1.5 = 4.5 NMR, the order will be active for 3 x 2 = 6 rounds)

  6. Duplicated purchase is not allowed, this also applies to active multi-round orders. If a buyer bought a subscription of 2 rounds, they can only make a new order for that product after the order completes in 2 weeks.

See forum post for full details


CoE Oct 2021 Transaction Log

-36 Numerbay

-8.8888 Newsletter

-34 Numerbay

-48 Numerbay

-36 Numerbay


CoE Updates

[Proposal] Writing Medium Article(or guest writing for the Newseltter) A Numerbay Walkthrough and How To Guide for buyers


CoE Fireside Chat with Richard and Anson Q4 2021

Slido Question Timestamps:

[0:00] Intro

[01:11] How is the progress on true contribution? It is slow but this quarter it is a huge focus

[03:57] I was wondering if you had any update to share with us regarding Numerai switching to 506(c)? SEE ABOVE*

[07:45] Did you guys see any improvement on the Meta Model Sharpe Ratio (or other metrics) since the release of Signals? And the (too early to say) new Dataset? Too early to say

[15:42] If NMR price declines substantially, but the meta model and hedge fund performs well, how will Numerai incentivise good predictions? NMR is up pretty substantially over the last 18 months or so from around $3 to $45. Its pretty volatile. Most people are not super sensitive to the volatility. Borrow NMR to hedge currency risk possibly.

[17:39] How reliable is the scoring code used for daily scores and payouts? Is there any way for participants to independently verify their scores after round close? System reliability such as being on time is still being worked on but is getting better. For trustworthiness, it is in our best interest to score correctly.

[21:17] What are Numerai's plans for incorporating Signals when making actual investment decisions? Or do you plan to use Signals submissions as data for classic? (Long Answer-See Video)

[26:33] Is it a problem that one user (Crowdcent) controls basically the whole Signals tournament with his stake using multiple but quite correlated models? Its not an optimal model with such high stakes but can be a good thing for NMR coming out of the pool. Will get better with more staked models.

[28:22] What is the delta between the top performers and the meta model? You can sometimes see in the sort run many models get to the top but come crashing down later. Its hard to say depending on period.

[29:50] Will you consider imposing a payout factor on Crowdcent? They are single handedly ruining payouts for everyone. NO

[31:26] How long will the old data format be available? Please say indefinitely. 2 answers. 1. We want to get rid of it but we are going to make feature sets. 2. Its going to be awhile though before we delete. We don’t want to break pipelines.

[34:33] Is leaderboard position and medals a good way for outside stakers to evaluate models to stake on? There is a lot to that question. Richard likes FNC but we are not sure. Medals might not be but depends on the length of time and the volatility. Some high ranked models are scary with low FNC.

[36:37] CoE was shaken quite a bit and also because of your role (or lack thereof). How happy are you with the impact the CoE has on the community? Hindsight insights? Richard likes Numerbay. Makes us think about how we can possibly support later. Things can get political. Could there be a leader? Not sure, that could be a bad idea too. Richard likes though its not run like a company. A lot more talking than action but its new. Anson actually likes the going back and forth of conflicts and resolutions. It needs time.

[43:01] Current MMC diagnostics underestimate the difficulty to get a good/stable MMC, any plans to improve it? Richard thinks they will delete MMC when True Contribution comes out. In the end its an estimate and its hard to project into the future. Richard thinks TC will be more stable. MMC is still at the mercy of feature exposure.

[46:00] Can we get Medals as NFTs? Alternative suggestion: Create limited edition NFTs which can be minted by burning medals. Anson loves it but not in our road map but could be in a hackathon etc. Cool idea came up last hackathon could be like making every model an NFT and it could give the owner all the rights. We will look into it more.

[47:45] Please keep supporting the old format data features for years. Excellent 2019-2020 models can be compared alongside the fund returns. This is very important! We do like the old models and data but we think the new data will be even better.

[49:00] Any update to instant early partial NMR unstaking based on max loss for rounds? Patrick is working on the staking and unstaking experience but we are thinking about different ways we can do different things. Not soon yet.

[49:42] Doesn’t it make sense to have the test set for future data release validation of results? Richard says it it could but he does not think we need the test set.

[51:25] Would new releases just be adding columns? YES hopefully. We are hopeful we wont need to delete columns

[54:05] Updates to stake management? Soon

[54:49] Is the meta model still stake weighted? Yes

[55:12] With the introduction of True contribution, will the 300k NMR stake cap (soft) be removed? Can you give a sneak into TC? Anson…interesting. Richard says its a possibility but we don’t know enough yet. There is limits but we will look into it.

[56:21] Wen resolve as of Wednesday's market close? Richard thinks ultimately this will go away and predictions will just be daily.

[57:57] Who's getting married next? Long pause…maybe Dennis…we will let you all know :)

[58:24] What part of Numerai (either the fund, the classic tournament, or the signals tournament) do you think needs the most improvement? Anson says Signals needs to grow a lot more.

[1:01:41] Were there any reasons that the specific validation eras were chosen? Any particular details that make them the best ones to validate our models on? Anson says they contains some exciting times such as $GME, china crackdowns etc. Also they are recent times.

[1:02:50] Is it possible to indicate on the leaderboard rankings (and api) which model uses legacy or the newer dataset? It would nice to see the impact of the new data Anson says unfortunately he does not think its possible. End points are different but we cant really tell who 100% for sure has migrated. IDs are the same.

[1:04:11] Do you think M Oliver is participating in the DataCrunch tournament? NO we don’t think so. Michael Oliver says “why would I waste my time”

[1:04:35] Numerai not owning the classic models is a cornerstone of the tournament setup. Any long-term plans to change this fundamental aspect? Anson no I don’t think so. We don’t own them its for you.

[1:05:15] Yo anson bro, wen stake management? Soon

[1:05:22] Congratulations on the performance of the Fund! Is there a way for us to see the neutralised targets for signal? Could numerai offer them? E.g, target_neut_10% We do give out neutralized targets for signals. Maybe we could later do it for different neutralization levels.

[1:06:45] Any plans to publicise Numerbay? Ask the CoE

[1:06:59] What plans do you have to increase participation in Signals? YES 1. Signals creation UX needs to be better. 2. Website basics and on-boarding. 3. Incentives

[1:08:00] Will we migrate to discord? If so, wen discord? Liam has been pushing for it. No definite plans yet. We love the functional programing language. Could come later.

[1:09:25] Crowdcent performance in signals is awful. Are you sure you want them to carry such a high stake-weight? BURN. We will let Crowdcent answer that in rocketchat.

[1:09:48] Site performance, scoring pipelines and other engineering functions have not been very reliable lately. Where is the point of failure? One point is the data vendors. Another point is putting pressure on engineers.

[1:10:43] Is outside staking helping the metamodel? Only as much as it helps the modelers express confidence in their models. We want them to have more access to capital.

[1:11:50] Wen tournament data targets release? Not sure

[1:12:09] Wen 2xCorr and 3xMMC on classic? Not anytime soon as we continue to work on TC

[1:12:23] When do you plan to go fully on-chain? Partially on-chain is creating a lot of confusion in terms of tracking. This is Ansons dream. Next year we want to be all on chain.

[1:15:43] The outage to rocket chat showed that if we continue using it as first line support then it should be monitored 24/7 as critical infrastructure. Anson says that’s true and maybe that could be a reason to switch to Discord

[1:16:05] Considering ETH's fees on L1 will remain high forever, will Numerai move to an L2 or will it become a multichain/multi-L2 dapp? Multi-chain maybe down the road. We are looking at different options still.

[1:17:34] When will it be possible to only send live submissions instead of the full tournament submissions? It is not possible. Anson says they decoupled live and test. Maybe when we give out the test targets. We will give test targets soon. Possibly in Dec 2021.

[1:18:40] Previously features have been deprecated suddenly, how long will the old datasets be available and are you willing to consider keeping them indefinitely? Answered previously

[1:18:53] How desperate do you need new investors? Have you had a real interest after talking about the fund performance? Richard says we have got a lot of new interest.

[1:20:04] If you say 'black box' you have already lost. Do you agree? Disagree in general

[1:23:15] Are there plans for better stake management? For example, I want to pay out every profit above a NMR threshold, or redistribution between models The reason why its taking so long for stake management is because the accounting back-end is not done super well. This is where we are working on the most. We are building a better accounting back-end. Give us time. Its coming soon.

[1:24:50] Why are gains reported by the Hedge Fund of the last 2 years (19%), so much less than the NMR gains of the average staking data scientist? Or are they the same? Its because we have a huge pile of NMR and we are giving it out aggressively. Its not the same as our hedge fund. Its always going to be possible to make more than the fund.

[1:26:04] Wen 5x MMC only staking in classic? We most likely wont be changing MMC as we work on TC.

[1:26:17] Wen "UNC" column in Signals so we can compare un-neutralized scores against models? Richard likes this. Its worth exploring.

[1:27:27] I am really curious about when you guys think the new dataset is a 'success', how good are you hoping it will perform (corr/mmc/return/...) We sort of responded to this a bit. Its showing signs already of being successful.

[1:28:00] So, when will be the 3000 features expansion?? December 2021

[1:28:10] How often do banks and financials enter the portfolio? Should we even be predicting on them, given how dissimilar they are to every other industry? Richard says it depends on the metric being used. There is financials in the data. Some sectors and companies in other countries run different so clusters could be good. Maybe categorical data etc.

[1:30:18] Being frank, GraphQL is awful. Would you consider a different system? Yes we will consider. Anson suggest letting him know what we like in the rocket.chat.

[1:31:05] Is the Meta Model getting less correlated to and better than the example model? Richard says yes and the new data could make it even better. Its possible it could get a lot better.

[1:31:56] What are your plans for Erasure(Bay)? There is no way to use it without an existing Authereum account. It would add much value to NMR, IMHO. Richard thinks it would be fun to have it working again. Maybe even work with metamask. Gas could be an issue. Wont be online anytime soon but we are not forgetting about it. Maybe when L2 comes.

[1:33:02] If we earn, on average, 2% for a certain round. This means more NMR in circulation (payouts from vault) -> deflation on NMR value, correct? Richard says not really. If most reinvest in the model the payouts are not entering the circulating supply. It looks healthy right now. Way more NMR is entering than leaving.

[1:34:39] Could you please give us the ability to download the list of ineligible tickers from signals? Anson will look into it.

[1:35:11] How does Signal MModel translate to fund positions? That MM is dominated by CrowdCent (concentration risk), what safeguards/constraints are there to mitigate Already talked about this.

[1:35:27] Wen will the fund investment come in the form of NMR? How $1bn in Numerai the company will affect $NMR? We sort of talked about this. Richard says the fund takes in Dollars and trade stocks.

[1:36:59] Why can't the investors (in the fund) be required to pay a portion of Fees in NMR which goes into treasury. Anson says interesting. Most institutional investors in hedge funds don’t want to touch crypto yet. We will think about that.

[1:37:33] What role are VCs playing in the enhancement of the Fund or increasing investment in the Fund? Or are they just leeching at NMR and looking for an exit? (both laugh) We have investors in the fund and the coin. The ones who invest in the company care about the company and the ones who invest in the coin care about the coin.

[1:39:30] Not asking that it would or should, but is Numerai legally capable of buying NMR from the open market? Richard says it thinks we cannot in the fund but the company could if it needed too. We already buy ETH for gas fees etc.


Other News

16GB Intermediate solution: XGB Era Boosting

Gradient Boosting Machines for multi-target regression

How to upload new dataset submissions through the API

[Beta Release] : Classics tournament reporting

Historical prices for 10,900 tickers - ticker map and downloader for eodhistoricaldata


Disclaimer: This is not an official Numerai newsletter. It is sponsored by the Numerai CoE, a decentralized autonomous organization. Every effort is made to provide accurate and complete information but there is no claims, promises or guarantees about the accuracy of the contents.