Symbolic Regression and Feature Engineering

[BradKML]

There is something funny that itched in my head when it comes to "feature engineering": realizing that a lot of the times it feels like banging rocks together if Exploratory Data Analysis (EDA) is too hard or annoying to deal with, AutoML seemed to be a simple solution, but it usually takes too long. Here are some examples:

• Microsoft phased out their open AutoML tooling https://github.com/microsoft/nni
• Alteryx has a good suite (same people behind EvalML) https://github.com/alteryx/featuretools
• https://github.com/mljar/mljar-supervised
• Standalone libraries exist https://github.com/feature-engine/feature_engine
• NVidia tried to beat Microsoft https://github.com/NVIDIA-Merlin/NVTabular https://github.com/NVIDIA-Merlin/Merlin
• Google too https://github.com/google/temporian
• Another toolset https://github.com/AutoViML/featurewiz

The field of Symbolic Regression seems to help this task. Originally, it was created for generating better equations, instead of using neural networks for predictions. Elegance is key, but it can also be used for tabular data.
Some of the libraries include (assuming we exclude quants):

• Python and Julia https://github.com/MilesCranmer/PySR
• People still uses Fortran (!?) https://github.com/rouyang2017/SISSO
• Legacy libraries for reference https://github.com/Ambrosys/glyph
• Scikit layer https://github.com/heal-research/pyoperon
• AutoML (thankfully) https://github.com/hftsoi/symbolfit
• https://github.com/MaxHalford/xgp

There are like 3 different approaches to the solution (and paobably all of them need to get benchmarked by SRBench https://github.com/cavalab/srbench):

1. LLM-based solutions (most popular now, least explored)

• https://github.com/deep-symbolic-mathematics/LLM-SR
• https://github.com/deep-symbolic-mathematics/TPSR
• https://github.com/mojivalipour/symbolicgpt

2. Genetic programming (most explored, more popular in the past)

• https://github.com/EMI-Group/evogp
• https://github.com/lagodiuk/genetic-programming
• https://github.com/cfusting/fast-symbolic-regression
• https://github.com/andreikitaitsev/SymINDy https://github.com/cavalab/ellyn

3. "Neuro-symbolic" (least popular, not as explored) and neural networks

• https://github.com/SymposiumOrganization/NeuralSymbolicRegressionThatScales
• https://github.com/martius-lab/EQL
• https://github.com/samuelkim314/DeepSymReg

The goal of OpenEvolve in this case is

1. Accelerate Symbolic Regression optimizers based on SRBench, potential future benchmarks, as well as whatever the user wants to throw at it

• https://github.com/omron-sinicx/srsd-benchmark
• https://github.com/CP3-Origins/cp3-bench

2. Gather tabular dataset from OpenML and see if Symbolic Regression can be turned into a feature engineering tool that enhance model prediction, while minimizing computational complexity and time cost (assuming that AutoML is also an option)

• https://github.com/openml/automlbenchmark
• https://github.com/Alex-Lekov/AutoML-Benchmark
• https://github.com/Ennosigaeon/automl_benchmark
• https://github.com/fabianmax/ML-Automation