当前位置：网站首页>Fisher linear discriminant analysis

Fisher linear discriminant analysis

2022-07-19 03:39:00 【elkluh】

Fisher Linear discriminant analysis is a linear classification method , Its main idea is ： Is that the variance within the class is small , There is a large difference between the class mean .（ Large between classes , Small in class ）

Take the classification of two classes as an example ：

Set the two classes from x1,x2 Project onto vector u On , In this way, from two-dimensional to one-dimensional , Then separate the two categories from the middle of the two groups .

If you want to make the difference between classes large , Then the average of each class will also differ greatly , set up $\mu_1,\mu_2$ Are the average values of plus and minus points respectively , So after projection , The square of their distance , That is to say $||\mu_1^T\mu-\mu_2^T\mu||^2$ As big as possible .

If you want to make the intra class variance small , Then the two classes are projected onto a straight line （ vector ） After the , Their points are $\mu^T\Sigma_1\mu,\mu^T\Sigma_2\mu$ , Represents the projection of two covariances .

$\Sigma_1=( x-\mu_1)(x-\mu_1)^T$

$\Sigma_2=( x-\mu_2)( x-\mu_2)^T$

So their sum should be as small as possible $\mu^T\Sigma_1\mu+\mu^T\Sigma_2\mu$

So take the big ones as molecules , The small is the denominator , The whole of their division is that the bigger the better ,

$Lossfunction:J(\mu) = \frac{\mu^T(\mu_1-\mu_2)(\mu_1-\mu_2)^T\mu}{ \mu^T \Sigma_1 \mu+ \mu^T \Sigma_2 \mu }$

set up $S_B=(\mu_1-\mu_2)(\mu_1-\mu_2)^T$

$S_W=\Sigma_1^2+\Sigma_2^2$

be $J(\mu)=\frac{ \mu^TS_B\mu }{\mu^TS_W\mu}$

Yes J(u) To find the derivative , Then there are $\frac{ \2 S_B\mu(\mu^TS_w\mu)-2 S_W\mu(\mu^TS_B\mu) }{(\mu S_W \mu)^2}$

Let the derivative be equal to 0, obtain $S_B\mu = (\frac{\mu^TS_B\mu}{u^TS_W\mu})S_W\mu$

The in brackets can be replaced by a scaling value

be $\mu = \alpha S_W^{-1}S_B\mu=\alpha S_W^{-1}(\mu_1-\mu_2)(\mu_1-\mu_2)^T\mu$

because $(\mu_1-\mu_2)^T$ and $\mu$ It becomes constant after multiplication

So the final $\mu = \beta S_W^{-1}(\mu_1-\mu_2)$

Is the vector we want to project .

import matplotlib.pyplot as plt
import numpy as np

def gauss2D(x, m, C):
  Ci = np.linalg.inv(C)  # Find the inverse of a matrix 
  dC = np.linalg.det(C)  # Find the determinant of the matrix 
  num = np.exp(-0.5 * np.dot((x-m).T, np.dot(Ci,(x-m))))
  den = 2 * np.pi * (dC**0.5) # Calculate the density function of the matrix 

  return num/den

def twoDGaussianPlot(nx, ny, m, C):
  x = np.linspace(-6, 6, nx)
  y = np.linspace(-6, 6, ny)
  X, Y = np.meshgrid(x, y, indexing='ij')

  Z = np.zeros([nx,ny])
  for i in range(nx):
    for j in range(ny):
      xvec = np.array([X[i,j], Y[i,j]])
      Z[i,j] = gauss2D(xvec, m, C)
  
  return X, Y, Z

X = np.random.randn(200, 2)
C1 = np.array([[2,1],[1,2]])
C2 = np.array([[2,1],[1,2]])
m1 = np.array([0, 3])
m2 = np.array([3,2.5])
A = np.linalg.cholesky(C1)

Y1 = X @ A.T + m1
Y2 = X @ A.T + m2

plt.figure(1)
plt.scatter(Y1[:,0], Y1[:,1], c='c', s=4)
plt.scatter(Y2[:,0], Y2[:,1], c='m', s=4)

Xp, Yp, Zp = twoDGaussianPlot(40,50,m1,C1)
plt.contour(Xp, Yp, Zp, 5)

Xp2, Yp2, Zp2 = twoDGaussianPlot(40,50,m2,C2)
plt.contour(Xp2, Yp2, Zp2, 5)

uF = np.linalg.inv(C1 + C2)@(m1-m2)
print(uF)
#ax.arrow(0, 0, *(uF*10), color='b', linewidth=2.0, head_width=0.20, head_length=0.25)
plt.arrow(0, 0, *(uF), color='b', linewidth=2.0, head_width=0.30, head_length=0.35)

plt.axis('equal')
plt.grid()
plt.xlim([-6,6])
plt.ylim([-5,8])

plt.savefig('density graph.png')

yp1 = Y1 @ uF
yp2 = Y2 @ uF

plt.figure(2)
plt.rcParams.update({'font.size':16})
plt.hist(yp1, bins=40)
plt.hist(yp2, bins=40)
plt.savefig('histogramprojections.png')