In statistics, logistic regression, or logit regression, or logit model is a regression model where the dependent variable (DV) is categorical. So it is a classification algorithm.

Logistic regression:
$$0 \le h_𝜃(x) \le 1$$

$$h_𝜃(x) = g(𝜃^Tx)$$

$$g(z) = \frac{1}{1+e^{-z}}$$

The above is the figure of sigmoid function. In machine learning, sigmoid function and logistic function mean the same thing.

When z>=0, g(z)>=0.5 and when z<=0, g(z)<=0.5. So we can predict y=1 if h(x)>=0.5. In this way, we can use logistic regression to form a classifier.
$$𝜃^Tx \ge 0, h_𝜃(x)=g(𝜃^Tx) \ge 0.5, y=1$$

$$𝜃^Tx \lt 0, h_𝜃(x)=g(𝜃^Tx) \lt 0.5, y=0$$

### Cost Function

$$h_𝜃(x)= \frac{1}{1+e^{𝜃^Tx}}$$

$$Cost(h_𝜃(x), y)= f(n) = \begin{cases} -log(h(x)), & \text{if y=1} \\ -log(1 - h(x)), & \text{if y=0} \\ \end{cases}$$

$$Cost(h_𝜃(x), y)=-ylog(h(x))-(1-y)log(1-h(x))$$

$$J(𝜃) = \frac{1}{m} \sum_{i=1}^m Cost(h(x^{(i)}),y^{(i)})$$

$$J(𝜃) = \frac{1}{m} \sum_{i=1}^m [-y^{(i)}log(h(x^{(i)}))-(1-y^{(i)})log(1-h(x^{(i)}))]$$

The gradient of the cost is a vector of the same length as 𝜃 where the $j^{th}$ element (for j = 0,1,…,n) is defined as follows:

So we update 𝜃 in this way:
Repeat {

} (simultaneously update for every j = 0,…,n)

### Overfitting & Regularization

There are 2 options for addressing overfitting, one is to reduce the number of features, the other is regularization.

In regularization:

• Keep all the features, but reduce magnitude/values of parameters 𝜃.
• Works well when we have a lot of features, each of which contributes a bit to predicting y.

#### Note

Note that you should not regularize the parameter $𝜃_0$. That is to say, you should not be regularizing the first parameter in theta.

### Implementation

Now let’s implement the algorithm in Matlab.
First, suppose you have the following function to calculate simgoid:
sigmoid.m

Here is the code to compute initial cost and gradient:
costFunction.m

If You want to compute initial cost and gradient for regularized logistic regression, then take a look at this:
costFunctionReg.m