How to assess if there is a strong signal in your dirty data [Project]

I'm sharing this new tabular data diagnostic tool (Entropic Scree). It can be used to estimate these properties of your high-d, real-world, dirty dataset:

• The informational volume of the signal (i.e., helps you assess whether the signal is strong enough to survive the dataset's idiosyncratic volume).
• The overall signal-to-idiosyncratic volume ratio (SNR).
• The intrinsic rank.
• Provides an exploratory map that allows for the identification of decoupled sub-networks of variables.
• The linear sufficiency (i.e., does the dataset align with the linear assumptions of standard PCA?).

Instead of evaluating linear variance, rank order, or Euclidean distance like traditional PCA variants, this new method evaluates a transformed mutual information metric. Relative to these baselines, it is less reliant on strong parametric or distance assumptions, making it appropriate to apply more broadly.

It also serves as a practical diagnostic of the theory explored in the From Garbage to Gold framework, which describes when and why uncurated, error-prone data can be used directly to create accurate prediction models.

There is a preprint that presents the full technical details, and Python and R packages will be released soon. Right now though, the original function is already available in R (see Quick Start R Function Code below).

• Entropic Scree Preprint: https://doi.org/10.5281/zenodo.22028087
• Entropic Scree GitHub: https://github.com/tjleestjohn/entropic-scree
• From Garbage to Gold Preprint: https://arxiv.org/abs/2603.12288

Let me know how it goes if you give it a try... or if you have any questions or comments of course.

############ # Quick Start R Function Code. # To load the function, copy and paste the following into your R console, then hit enter. ############ # 1. Define the direct URL to the raw function script on GitHub url <- "https://raw.githubusercontent.com/tjleestjohn/entropic-scree/main/Entropic.Scree.v1.0.0%20-%20ENLI.R" # 2. Define what you want to name the file on your computer file_name <- "Entropic.Scree.v1.0.0 - ENLI.R" # 3. Download the script to your current working directory download.file(url, destfile = file_name) # 4. Source the core function into your R environment source(file_name) # 5. Ex. To run the function and extract bipolar modules: # results <- Entropic.Scree(dt # , extract_bipolar_modules = TRUE) # # View the extracted structural sub-networks for the primary axes: # results$bipolar_modules