close

Function Repository Resource:

FirstOrderCorrelation

Source Notebook

Compute the first order correlation matrix from an original correlation matrix

Contributed by: Seth J. Chandler

ResourceFunction["FirstOrderCorrelation"][m]

computes the first order correlation matrix of a matrix m.

Details and Options

The matrix must be invertible.
The matrix should be symmetric and positive definite in order for the results to be useful.
Setting the option "ConditionsCheck" to True will warn if the matrix can not be confirmed to be symmetric and positive definite.

Examples

Basic Examples (2) 

Compute the first order correlation of a rational 3×3 matrix:

In[1]:=
ResourceFunction[
 "FirstOrderCorrelation"][{{1, -3/5, 7/10}, {-3/5, 1, -21/50}, {7/10, -21/50, 1}}]
Out[1]=
Image

Compute the first order correlation of a symbolic symmetric 2×2 matrix:

In[2]:=
ResourceFunction["FirstOrderCorrelation"][
  Array[x, {2, 2}] /. x[i_, j_] :> x[j, i] /; i > j] // FullSimplify
Out[2]=
Image

Scope (1) 

FirstOrderCorrelation works on large matrices:

Image
In[3]:=
ResourceFunction["FirstOrderCorrelation"][
   randomMatrix]; // RepeatedTiming
Out[3]=
Image

Options (2) 

ConditionsCheck (2) 

Unless the "ConditionsCheck" option is set to True, FirstOrderCorrelation will evaluate and provide no warning even if the matrix is not symmetric:

In[4]:=
ResourceFunction[
 "FirstOrderCorrelation"][{{1, -4/5, 7/10}, {-3/5, 1, -21/50}, {7/10, -21/50, 1}}]
Out[4]=
Image
Image
Image
Out[4]=
Image

Setting "ConditionsCheck" to True also issues a warning if the matrix is not unambiguously positive definite and if it is not unambiguously positive semi-definite:

Image
Image
Image
Out[4]=
Image

Applications (2) 

Find the value of an element in a 3×3 zero-order correlation matrix that will set to 0 the corresponding element in the first order correlation matrix:

In[5]:=
Solve[0 == ResourceFunction[
    "FirstOrderCorrelation"][{{1, -3/5, 7/10}, {-3/5, 1, x}, {7/10, x,
       1}}][[2, 3]], x]
Out[5]=
Image

Find the element in a 4×4 zero-order correlation matrix that will set to 0 the corresponding element in the first order correlation matrix:

In[6]:=
Solve[0 == ResourceFunction[
    "FirstOrderCorrelation"][{{1, -3/5, -5/9, 7/10}, {-3/5, 1, -2/11, 3/10}, {-5/9, -2/11, 1, x}, {7/10, 3/10, x, 1}}][[3, 4]], x]
Out[6]=
Image

Properties and Relations (1) 

The first order correlation matrix of the first order correlation matrix is the same as the original matrix:

In[7]:=
Counts[Table[
  NestList[ResourceFunction["FirstOrderCorrelation"]/*Chop, Correlation[RandomVariate[NormalDistribution[0, 1], {100, 10}]], 3] // (#[[1]] - #[[3]] &)/*Chop, {100}]]
Out[7]=
Image

Possible Issues (1) 

The function will fail and throw a message if the matrix is singular:

Image
Image
Out[7]=
Image

Neat Examples (5) 

Compare the false positive rates and true positive rates among "minority" and "majority" populations when classifying a population where the first variable is correlated with the second, the first variable is correlated with the third, but, after the first variable is taken into account, the second variable is uncorrelated with the third.

Take a first order correlation matrix and compute the zero-order correlation matrix:

In[8]:=
m = ResourceFunction["FirstOrderCorrelation"][zeroOrderCorrelation, "ConditionsCheck" -> True]
Out[8]=
Image
In[9]:=
zeroOrderCorrelation = ResourceFunction["FirstOrderCorrelation"][m]
Out[9]=
Image

Use the zero-order correlation matrix as the covariance matrix of a multinormal copula and use that to produce 1000 instances of training data and 1000 instances of test data:

Image

Classify uses AutoML to figure out how to best classify the data; ClassifierMeasurements uses the trained classifier and the test data to compute various performance statistics:

In[10]:=
cl = Classify[Map[Most][training] -> Map[Last][training], TrainingProgressReporting -> None];
ClassifierMeasurements[cl, Map[Most][test] -> Map[Last][test], {"Accuracy", "AreaUnderROCCurve" -> 1, "FalsePositiveRate" -> 1, "TruePositiveRate" -> 1}] // AssociationThread[{"Accuracy", "AreaUnderROCCurve" -> 1, "FalsePositiveRate" -> 1, "TruePositiveRate" -> 1} -> #] &
Out[10]=
Image

Select picks out minority members from the test data set; the same performance statistics are generated:

In[11]:=
With[{minority = Select[test, #[[2]] == 1 &]}, ClassifierMeasurements[cl, Map[Most][minority] -> Map[Last][minority], {"Accuracy", "AreaUnderROCCurve" -> 1, "FalsePositiveRate" -> 1, "TruePositiveRate" -> 1}]
  ] // AssociationThread[{"Accuracy", "AreaUnderROCCurve" -> 1, "FalsePositiveRate" -> 1, "TruePositiveRate" -> 1} -> #] &
Out[11]=
Image

Select picks out majority members from the test data set; the same performance statistics are generated:

In[12]:=
With[{majority = Select[test, #[[2]] == 0 &]}, ClassifierMeasurements[cl, Map[Most][majority] -> Map[Last][majority], {"Accuracy", "AreaUnderROCCurve" -> 1, "FalsePositiveRate" -> 1, "TruePositiveRate" -> 1}]
  ] // AssociationThread[{"Accuracy", "AreaUnderROCCurve" -> 1, "FalsePositiveRate" -> 1, "TruePositiveRate" -> 1} -> #] &
Out[12]=
Image

Although there is no intentional "discrimination," and while the accuracy of the classifier is roughly the same for both groups, the false positive rates and true positive rates are higher for the majority than the minority.

Publisher

Seth J. Chandler

Version History

  • 1.0.0 – 29 March 2021

License Information