Functions you meet in ML: lines, powers, exp and log

Before this lesson: lesson 02 (functions, $\sum$, $\prod$, argmax) and lesson 03 (arrays and plots). You need school algebra: solve $2x + 3 = 7$. Time: about 55 minutes, plus the notebook.

---

Why this matters #

Every model in this site is built from these functions. Linear regression predicts $\hat y = wx + b$ (with many features, $\hat y = w^\top x + b$). The squared error $(y - \hat y)^2$ is a power. Logistic regression, softmax classifiers and every language model use $e^z$ to turn scores into probabilities (lesson 05). Training minimizes the negative log-likelihood (the cross-entropy loss), a sum of logs. Learning rates are searched on a log scale ($10^{-5}, 10^{-4}, \dots$), loss curves are plotted on log axes, and neural scaling laws (loss against model size) are straight lines on log-log plots.

1. Predict first #

2. The picture #

A graph is a set of points. The graph of a function $f$ is the set of all points $(x, f(x))$. The input $x$ is on the horizontal axis, the output $f(x)$ on the vertical axis. To read $f(2)$, go to $x = 2$ and up (or down) to the curve.

Lines. The graph of $f(x) = wx + b$ is a straight line. The number $w$ is the slope: when $x$ grows by 1, $y$ changes by $w$. For two points on the line, the slope is "rise over run": $w = \Delta y / \Delta x$. The number $b$ is the intercept: the height where the line crosses the vertical axis, because $f(0) = b$.

Interactive figure: loading…

Moving and scaling a graph. Many ML functions are a basic function that is moved or stretched. From $f$ we build $g(x) = a\,f\big(c(x - h)\big) + k$. Each number does one thing: $h$ moves the graph right, $k$ moves it up, $a$ stretches it up and down, $c$ squeezes it left and right. The figure lets you test each one.

Interactive figure: loading…

Exp and log are mirror images. The exponential $y = b^x$ (for a base $b > 0$, $b \ne 1$) takes every real $x$ to a positive $y$. Its inverse, the logarithm $\log_b$, goes back: $\log_b y$ is the power you need, so $b^{\log_b y} = y$. If $(x, y)$ is on the graph of $b^x$, then $(y, x)$ is on the graph of $\log_b$. Swapping the two coordinates is a mirror image across the line $y = x$.

Interactive figure: loading…

3. Definitions and main results #

3.1 Lines and affine functions #

A function $f(x) = wx + b$ is called affine (school books say "linear function"; in linear algebra, "linear" also needs $f(0) = 0$, so only $b = 0$ is linear; see Linear Algebra lesson 01).

Line through two points. Let $(x_1, y_1)$ and $(x_2, y_2)$ be two points with $x_1 \ne x_2$.

1. The slope is $w = \dfrac{y_2 - y_1}{x_2 - x_1}$ (rise over run). Use the same order of points above and below.
2. The line goes through $(x_1, y_1)$, so $y_1 = w x_1 + b$. Solve for $b$: $b = y_1 - w x_1$.

If $x_1 = x_2$, the run is 0 and the slope is undefined: the vertical line $x = x_1$ is not the graph of a function, because one input would have many outputs.

• $w > 0$: the line goes up from left to right; $w < 0$: it goes down; $w = 0$: it is flat.
• In ML, $\hat y = wx + b$ is a model with one feature $x$: $w$ is the weight and $b$ is the bias. With $n$ features, $\hat y = w_1 x_1 + \dots + w_n x_n + b = w^\top x + b$.

3.2 Moving and scaling graphs #

For $g(x) = a\,f\big(c(x - h)\big) + k$:

Change	Effect on the graph of $f$	Example with $f(x) = x^2$
$f(x - h)$, $h > 0$	moves right by $h$	$(x - 2)^2$: lowest point at $x = 2$
$f(x) + k$, $k > 0$	moves up by $k$	$x^2 + 1$
$a f(x)$, $a > 1$	stretches up and down by the factor $a$	$2x^2$ is narrower
$f(cx)$, $c > 1$	squeezes left and right by the factor $c$	$(2x)^2 = 4x^2$
$-f(x)$	mirror image across the $x$-axis	$-x^2$ opens downward
$f(-x)$	mirror image across the $y$-axis	$e^{-x}$ is $e^x$ mirrored

Why $f(x - h)$ moves right. The new graph at $x$ shows the old value at $x - h$. So the old point at $x_0$ appears where $x - h = x_0$, that is, at $x = x_0 + h$: to the right.

3.3 Powers #

For a number $a$ and a positive integer $n$, $a^n = a \cdot a \cdots a$ ($n$ factors). The rules come from counting factors:

Rule	Reason
$a^m a^n = a^{m+n}$	$m$ factors, then $n$ more factors
$(a^m)^n = a^{mn}$	$n$ groups of $m$ factors
$(ab)^n = a^n b^n$	reorder the $2n$ factors
$a^0 = 1$ ($a \ne 0$)	needed so that $a^m a^0 = a^m$
$a^{-n} = 1/a^n$ ($a \ne 0$)	needed so that $a^n a^{-n} = a^0 = 1$
$a^{1/2} = \sqrt a$ ($a \ge 0$)	needed so that $(a^{1/2})^2 = a^1 = a$

The last three lines are definitions: they are the only choices that keep the first rule true. For a non-integer power $a^{p}$ we assume $a > 0$ (for example, $(-8)^{1/2}$ is not a real number).

Non-rules. $(a + b)^2 \ne a^2 + b^2$ (the correct expansion is $a^2 + 2ab + b^2$), and $a^m + a^n \ne a^{m+n}$ (example: $2^1 + 2^2 = 6$, but $2^3 = 8$).

A polynomial is a sum of powers with coefficients, $p(x) = c_d x^d + \dots + c_1 x + c_0$; its degree is the largest power $d$ with $c_d \ne 0$. The graph of $x^2$ is a U-shaped curve (a parabola) with its lowest point at 0. This is the shape of the squared error $(y - \hat y)^2$: an error of 2 costs 4, an error of 10 costs 100, so large errors count much more. The graph of $x^3$ is increasing and changes sign at 0.

3.4 The exponential function #

For a base $b > 0$, the exponential function $x \mapsto b^x$ is defined for every real $x$ (the rules of 3.3 fix its values at fractions, and the graph fills in the rest without gaps).

• $b^x > 0$ for every $x$, and $b^0 = 1$.
• $b > 1$: exponential growth (the function increases). $0 < b < 1$: exponential decay (it decreases). $b = 1$: the constant 1.
• The rules of 3.3 hold for all real powers: $b^{x+y} = b^x b^y$, $(b^x)^y = b^{xy}$, $b^{-x} = 1/b^x$.

The number $e$. The most used base is Euler's number $e \approx 2.71828$. One way to define it: $e$ is the limit of $(1 + 1/n)^n$ as $n$ grows (interest of $100\,\%$ per year, paid in $n$ small parts). We write $\exp(x) = e^x$. Why this base? The graph of $e^x$ has slope exactly $e^x$ at every point; at $x = 0$ the slope is 1. This makes derivatives simple (lesson 07 shows it).

Exponential growth is fast. Each step of 1 in $x$ multiplies $e^x$ by $e$. So $e^{10} \approx 22\,026$, $e^{100} \approx 2.7 \times 10^{43}$, and $e^{710}$ is larger than the largest float64 number (about $1.8 \times 10^{308}$): np.exp(710) returns inf. Lesson 05 shows how ML code avoids this.

Doubling time. If $y(t) = y_0 e^{rt}$ with rate $r > 0$, then $y$ doubles every $T_2 = \log 2 / r \approx 0.693 / r$ time units. Reason: $y_0 e^{r(t + T_2)} = y_0 e^{rt} e^{rT_2}$, and $e^{rT_2} = 2$ exactly when $rT_2 = \log 2$ (the log is defined next).

3.5 The logarithm #

Definition. Let $b > 0$, $b \ne 1$. For $y > 0$, the logarithm to base $b$, $\log_b y$, is the number $x$ with $b^x = y$. In words: "which power of $b$ gives $y$?"

• $\log_2 8 = 3$ because $2^3 = 8$. $\log_{10} 1000 = 3$. $\log_2(1/8) = -3$ because $2^{-3} = 1/8$.
• Inverse pair: $\log_b(b^x) = x$ for every real $x$, and $b^{\log_b y} = y$ for every $y > 0$.
• $\log_b 1 = 0$ (because $b^0 = 1$) and $\log_b b = 1$.
• Domain: $\log_b y$ exists only for $y > 0$, because $b^x$ is never 0 or negative. For $0 < y < 1$ and $b > 1$, $\log_b y < 0$; as $y$ goes down to 0, $\log_b y$ goes to $-\infty$.
• For $b > 1$, $\log_b$ is strictly increasing: $0 < p < q$ implies $\log_b p < \log_b q$. (If $\log_b p \ge \log_b q$, apply the increasing function $b^x$ to both sides: $p \ge q$.)

Notation. In this site, as in ML papers and in NumPy, $\log$ without a base is the natural logarithm, base $e$ (np.log). School books write $\ln$. We write $\log_2$ (np.log2) and $\log_{10}$ (np.log10) when we need another base.

The logarithm grows slowly: $\log_{10}$ of a million is only 6. For a whole number $y \ge 1$, $\lfloor \log_{10} y \rfloor + 1$ is its number of digits. For $y = 2^n$, $\log_2 y = n$ is the number of times you halve $y$ to reach 1.

3.6 The log rules, and why they are true #

Let $a, c > 0$ and write $a = b^u$, $c = b^v$, so $u = \log_b a$ and $v = \log_b c$. Each log rule is a power rule read backwards:

1. Product: $ac = b^u b^v = b^{u + v}$, so $\log_b(ac) = \log_b a + \log_b c$.
2. Quotient: $a/c = b^{u - v}$, so $\log_b(a/c) = \log_b a - \log_b c$.
3. Power: $a^k = (b^u)^k = b^{ku}$, so $\log_b(a^k) = k \log_b a$ for every real $k$.
4. Change of base: start from $y = b^{\log_b y}$ and take the natural log of both sides. Rule 3 gives $\log y = \log_b y \cdot \log b$, so $\log_b y = \dfrac{\log y}{\log b}$. Every log is the natural log times a constant. Example: $\log_2 10 = \log 10 / \log 2 \approx 2.303 / 0.693 \approx 3.32$.
5. Every exponential is a power of $e$: $b^x = e^{x \log b}$, because $e^{\log b} = b$.

Non-rules (common errors):

• $\log(a + b) \ne \log a + \log b$. Counterexample $a = b = 1$: $\log 2 \approx 0.693$, but $\log 1 + \log 1 = 0$.
• $\log(ab) \ne \log a \cdot \log b$, and $\log a / \log b \ne \log(a/b)$ (in fact $\log a / \log b = \log_b a$).
• $(\log a)^k \ne k \log a$; only $\log(a^k) = k\log a$.

3.7 Why ML loves logs #

(a) The log does not move the maximum. Let $L(\theta) > 0$ for all parameters $\theta$ (for example a likelihood, the probability of the data under the model with parameters $\theta$).

Claim. $\arg\max_\theta \log L(\theta) = \arg\max_\theta L(\theta)$.

Proof. Because $\log$ is strictly increasing, for any two parameters $\theta_1, \theta_2$: $L(\theta_1) \ge L(\theta_2)$ holds exactly when $\log L(\theta_1) \ge \log L(\theta_2)$. So "$\theta^\star$ is at least as good as every $\theta$" means the same thing for $L$ and for $\log L$. $\blacksquare$

The maximum value changes ($\log L(\theta^\star)$ instead of $L(\theta^\star)$), but the location $\theta^\star$ does not. The same proof works for any strictly increasing function, and for argmin.

(b) A product becomes a sum. For independent data points, the likelihood is a product $L = \prod_{i=1}^{m} p_i$, where $p_i$ is the probability of point $i$. By the product rule,

Sums are easier to differentiate and to average, and they are safe on a computer. The product of 1000 numbers equal to 0.01 is $10^{-2000}$. The smallest positive float64 number is about $5 \times 10^{-324}$, so the product underflows: the computer stores 0.0. The log-likelihood is $1000 \log 0.01 \approx -4605.17$, a normal number. Training minimizes the negative log-likelihood $-\sum_i \log p_i$.

(c) Log scales show many orders of magnitude. A learning rate of $10^{-5}$ and one of $10^{-1}$ differ by a factor $10^4$. On a log axis they are 4 equal steps apart, and every factor 10 in between gets the same space.

3.8 Log-scale plots #

On a log axis, the distance between two values is the difference of their logs. Ticks 1, 10, 100, 1000 are equally spaced: equal distances mean equal ratios, not equal differences.

• Semilog plot (log $y$, linear $x$). If $y = a b^x$, then $\log y = \log a + x \log b$. This is a line in the variables $(x, \log y)$, with slope $\log b$ and intercept $\log a$. So exponential growth or decay is a straight line on a semilog plot.
• Log-log plot (log $y$ and log $x$). If $y = a x^k$ (a power law), then $\log y = \log a + k \log x$: a straight line in $(\log x, \log y)$ with slope $k$, the exponent. The slope does not depend on the base of the log, as long as both axes use the same base.

Interactive figure: loading…

3.9 Exponentials beat polynomials, but maybe late #

Fact. For every base $b > 1$ and every power $k > 0$, $b^x / x^k$ grows without limit as $x$ grows. So $b^x$ is eventually larger than $x^k$, however large $k$ is.

A plot on a small range can hide this. For $x > 0$, the two curves $2^x$ and $x^{10}$ cross twice: $2^x > x^{10}$ on $(0, 1.0776)$, $2^x < x^{10}$ on $(1.0776, 58.77)$, and $2^x > x^{10}$ for every $x > 58.77$. At $x = 30$, $2^{30} \approx 1.07 \times 10^9$ while $30^{10} \approx 5.9 \times 10^{14}$; at $x = 100$, $2^{100} \approx 1.27 \times 10^{30}$ while $100^{10} = 10^{20}$.

Interactive figure: loading…

Why exponentials win: a short argument

Take logs: $\log\big(b^x / x^k\big) = x \log b - k \log x = x\Big(\log b - k\,\dfrac{\log x}{x}\Big)$. The ratio $\log x / x$ goes to 0 as $x$ grows. One way to see it: $\log x < \sqrt{x}$ for all $x > 0$ (the difference $\sqrt x - \log x$ is smallest at $x = 4$, where it is $2 - \log 4 \approx 0.61 > 0$; finding this minimum uses a derivative, lesson 07). So $0 < \log x / x < 1/\sqrt{x}$ for $x > 1$, which goes to 0. Then the bracket tends to $\log b > 0$, the product with $x$ grows without limit, and so does $b^x / x^k = e^{(\text{that product})}$.

In computer science this is the gap between polynomial and exponential cost. A grid search over $d$ hyperparameters with 10 values each needs $10^d$ runs: this grows exponentially in $d$.

Answers to section 1. (1) At $x = 30$, $x^{10} \approx 5.9 \times 10^{14}$ is much larger than $2^{30} \approx 1.07 \times 10^9$. At $x = 100$, $2^{100} \approx 1.27 \times 10^{30}$ beats $100^{10} = 10^{20}$. The exponential wins for every $x > 58.77$. (2) It prints 0.0: the true value $10^{-2000}$ underflows. The sum of the logs, $-4605.17$, is fine. (3) No. Counterexample $a = b = 1$: $\log 2 \ne 0$. The true rule is about products: $\log(ab) = \log a + \log b$.

4. Worked example #

(a) A line from two points. Find the line through $(1, 2)$ and $(3, 8)$.

1. Slope: $w = \dfrac{8 - 2}{3 - 1} = \dfrac{6}{2} = 3$.
2. Intercept: $b = y_1 - w x_1 = 2 - 3 \cdot 1 = -1$.
3. The line is $y = 3x - 1$. Check the second point: $3 \cdot 3 - 1 = 8$ ✓.
4. Use it: at $x = 2$ the model predicts $3 \cdot 2 - 1 = 5$.

(b) Solve for a power.

• $2^x = 32$. Since $32 = 2^5$, $x = \log_2 32 = 5$.
• $3 \cdot 2^x = 100$. Divide by 3: $2^x = 100/3$. Take $\log_2$: $x = \log_2(100/3) = \dfrac{\log(100/3)}{\log 2} = \dfrac{3.5066}{0.6931} \approx 5.059$. Check: $3 \cdot 2^{5.059} \approx 3 \cdot 33.33 = 100$ ✓.

(c) Likelihood and log-likelihood of coin flips. A coin shows heads with probability $p$. We see H, H, T, H (independent flips). The likelihood is the product of the four probabilities:

$p$	$L(p)$	$\log L(p)$
$0.5$	$0.5^3 \cdot 0.5 = 0.0625$	$4 \log 0.5 \approx -2.773$
$0.75$	$0.75^3 \cdot 0.25 \approx 0.1055$	$3\log 0.75 + \log 0.25 \approx -0.863 - 1.386 = -2.249$
$0.9$	$0.9^3 \cdot 0.1 = 0.0729$	$3 \log 0.9 + \log 0.1 \approx -0.316 - 2.303 = -2.619$

Both columns are largest at $p = 0.75$ (in the notebook you check a fine grid of $p$). The maximizer $3/4$ is the fraction of heads. (Lesson 07 finds it exactly by setting the slope of $\log L$ to 0: $3/p - 1/(1-p) = 0$ gives $p = 3/4$.)

(d) Read an exponential from a semilog plot. Data lie on a straight line on a semilog plot and pass through $(0, 3)$ and $(5, 96)$. So $y = a b^x$.

1. At $x = 0$: $y = a b^0 = a$, so $a = 3$.
2. At $x = 5$: $3 b^5 = 96$, so $b^5 = 32$ and $b = 32^{1/5} = 2$.
3. The data follow $y = 3 \cdot 2^x$: $y$ doubles with each step of 1 in $x$. On a base-10 semilog plot the slope is $\log_{10} 2 \approx 0.301$, as in the figure of section 3.8.

5. In code #

NumPy checks the worked example. The notebook for this lesson has these cells and the exercises.

import numpy as np
from scipy.optimize import brentq

# (a) line through two points
x1, y1, x2, y2 = 1.0, 2.0, 3.0, 8.0
w = (y2 - y1) / (x2 - x1)
b = y1 - w * x1
print(w, b, w * 2 + b)                     # 3.0 -1.0 5.0

# (b) solve 3 * 2**x = 100
x = np.log(100 / 3) / np.log(2)
print(round(x, 3), 3 * 2**x)               # 5.059 100.00000000000006

# log rules: True, and the non-rule: False
a, c = 7.0, 0.3
print(np.isclose(np.log(a * c), np.log(a) + np.log(c)))   # True
print(np.isclose(np.log(a + c), np.log(a) + np.log(c)))   # False

# (c) the product underflows, the sum of logs does not
p = np.full(1000, 0.01)
print(np.prod(p), np.sum(np.log(p)))       # 0.0 -4605.17018598809

# 2**x = x**10: compare logs to avoid huge numbers, then find the two crossings
g = lambda x: x * np.log(2) - 10 * np.log(x)
print(brentq(g, 1.01, 2), brentq(g, 10, 100))   # 1.0775501500027171 58.77010593792138

# (d) slope of exponential data on a semilog plot
xs = np.arange(0, 6.0)
ys = 3 * 2**xs
slope, intercept = np.polyfit(xs, np.log10(ys), 1)
print(round(slope, 4), round(10**intercept, 4))  # 0.301 3.0

The function np.log is the natural log. np.log(0) returns -inf and np.log(-1) returns nan, both with a warning: the log of a number $\le 0$ is not defined.

6. Where it shows up #

• Linear models. Linear regression $\hat y = w^\top x + b$; one neuron computes an affine function $w^\top x + b$ before its activation. The slope says how much the prediction changes per unit of a feature.
• Losses. The squared error $(y - \hat y)^2$ (a power) in regression; the negative log-likelihood $-\sum_i \log p_i$, also called the cross-entropy loss, in classification and in language models (link: the /information-theory/ track explains cross-entropy).
• The exponential inside models. Sigmoid and softmax use $e^z$ (lesson 05); the Gaussian density uses $e^{-x^2/2}$ (lesson 08); exponential learning-rate decay $\eta_t = \eta_0 \gamma^t$; the discount factor $\gamma^t$ in reinforcement learning; exponential moving averages in the Adam optimizer.
• Log scales. Learning-rate and regularization searches on a log grid; loss curves on log axes; scaling laws: the test loss of language models falls as a power law of model size, data size and compute, so it is a straight line on a log-log plot (Kaplan et al., 2020).
• Log transforms of data. Skewed positive features (prices, incomes, counts) are often replaced by $\log x$ or $\log(1 + x)$ before training, so that factors become differences.
• ML for science. Radioactive decay and drug concentration (exponential decay); the Arrhenius law of reaction rates $k = A e^{-E_a / (RT)}$; the pH and decibel scales are logarithms; power laws in turbulence and in network degree distributions are read on log-log plots.

7. Common mistakes #

• $\log(a + b) = \log a + \log b$. False. The log turns products into sums, not sums into sums.
• Taking the log of 0 or of a negative number. The log is defined only for inputs $> 0$. In code, a probability that rounds to 0 gives -inf; add a check, or compute the log directly in a stable way (lesson 05).
• Mixing up bases. np.log is the natural log ($\ln$). np.log10 and np.log2 are other bases; math.log(x, b) takes a base. Results in different bases differ by a constant factor.
• Moving the graph the wrong way. $f(x - 2)$ moves right, $f(x + 2)$ moves left.
• Slope with mixed order. In $(y_2 - y_1)/(x_2 - x_1)$, use the same point first above and below; $(y_2 - y_1)/(x_1 - x_2)$ has the wrong sign.
• $(a + b)^2 = a^2 + b^2$. False; the cross term $2ab$ is missing.
• Reading a log axis as linear. On a log axis, the middle between 1 and 100 is 10, not 50.
• Python precedence. -2**2 is -4 in Python (the power comes first), and (-2)**2 is 4.
• Judging growth from a small range. On $[0, 30]$, $x^{10}$ looks faster than $2^x$; on $[0, 100]$, it is not.

8. Check yourself #

1. Find the line through $(-1, 4)$ and $(2, -2)$. Where does it cross the $y$-axis?
   Answer

   $w = (-2 - 4)/(2 - (-1)) = -6/3 = -2$. Then $b = y_1 - w x_1 = 4 - (-2)(-1) = 2$. The line is $y = -2x + 2$; it crosses the $y$-axis at $(0, 2)$. Check: $-2 \cdot 2 + 2 = -2$ ✓.
2. Compute without a calculator: $\log_2 64$, $\log_{10} 0.001$, $\log(e^5)$, $e^{\log 7}$.
   Answer

   $6$ (since $2^6 = 64$), $-3$ (since $10^{-3} = 0.001$), $5$, and $7$: exp and log undo each other.
3. Write $\log\big(x^2 y / z\big)$ as a sum and difference of $\log x$, $\log y$, $\log z$ ($x, y, z > 0$).
   Answer

   $2\log x + \log y - \log z$: product rule, quotient rule, then the power rule for $x^2$.
4. A quantity grows as $y(t) = 5 e^{0.05 t}$. What is its doubling time? How long until it is 10 times larger?
   Answer

   Doubling: $e^{0.05 T} = 2$, so $T = \log 2 / 0.05 \approx 13.86$. Factor 10: $T = \log 10 / 0.05 \approx 46.05$. The starting value 5 does not matter.
5. Data on a log-log plot lie on a line through $(\log_{10} x, \log_{10} y) = (0, 1)$ and $(2, 5)$. Give the formula for $y$.
   Answer

   Slope $k = (5 - 1)/(2 - 0) = 2$, intercept $\log_{10} a = 1$, so $a = 10$. So $y = 10x^2$. Check: $x = 100$ gives $10 \cdot 10^4 = 10^5$, and $\log_{10} 10^5 = 5$ ✓.
6. Your code computes np.prod(probs) for 5000 probabilities around 0.3 and gets 0.0. Is the model wrong? What should you compute instead?
   Answer

   Not necessarily: $0.3^{5000} \approx 10^{-2614}$ underflows to 0 in float64. Compute the log-likelihood `np.sum(np.log(probs))` instead; it is about $-6020$ and has the same maximizer.

Recall cardsAnswer in your head first, then reveal. Add the cards to your review deck to see them again just before you would forget.

1. How do you get the slope of the line through $(x_1, y_1)$ and $(x_2, y_2)$?

   Show answer

2. Compute the slope of the line through $(1, 2)$ and $(3, 8)$.

   Show answer

3. In which direction does $f(x - 2)$ move the graph of $f$, and why?

   Show answer

4. Why is $a^0 = 1$ and $a^{-n} = 1/a^n$ (for $a \ne 0$)?

   Show answer

5. Define $\log_b y$, and give its domain.

   Show answer

6. State the product, power and change-of-base rules of the logarithm.

   Show answer

7. Why does maximizing $\log L(\theta)$ give the same $\theta$ as maximizing $L(\theta)$?

   Show answer

8. What does Python give for the product of 1000 numbers equal to 0.01, and what is the fix?

   Show answer

9. What does a straight line on a semilog plot (log $y$, linear $x$) mean? And on a log-log plot?

   Show answer

10. What is the doubling time of $y_0 e^{rt}$?

    Show answer

Add these cards to my reviewOpen the review →

9. Interview angle #

• "Why do we train with the log-likelihood?" — The log is strictly increasing, so the maximizer is the same. The product of many probabilities becomes a sum, which does not underflow, is easy to differentiate, and splits into one term per example (so mini-batches work).
• "What does a straight line on a log-log plot tell you?" — A power law $y = ax^k$ with exponent equal to the slope. Example: neural scaling laws for loss against parameters, data and compute.
• "When would you log-transform a feature or a target?" — When it is positive and skewed, or when effects are multiplicative: after the log, a factor becomes a difference, and a model with additive effects fits better. Use $\log(1 + x)$ when zeros occur.
• "Why are learning rates searched on a log grid?" — Good values differ by orders of magnitude; equal steps in $\log \eta$ give each factor of 10 the same number of trials.

10. Practice #

The notebook for this lesson runs in Colab or in your browser. It has five exercises, each with a check cell and a folded solution:

1. Write line_through(p, q) that returns $w$ and $b$; check it on a third point of the line.
2. Check the log rules on random positive numbers, and find a counterexample to $\log(a + b) = \log a + \log b$.
3. See the product of 1000 probabilities underflow, and compute the log-likelihood as a sum of logs instead.
4. Find both crossings of $2^x$ and $x^{10}$ with bisection, written by you.
5. Recover the growth rate of noisy exponential data with np.polyfit on $\log y$.

11. Watch and read #

What's so special about Euler's number e? | Chapter 5, Essence of calculus3Blue1Brown · Essence of calculus, chapter 5 · 14 min

Why the slope of eˣ equals eˣ, and how every exponential bˣ is e to the power x·log b.

Logs (logarithms), Clearly Explained!!!StatQuest with Josh Starmer · 16 min

The log rules and why statistics and ML work on the log scale.

Exponential growth and epidemics3Blue1Brown · 9 min

Exponential growth on linear and log axes, and why it must slow down in the real world.

• Jay Abramson et al., Algebra and Trigonometry 2e (OpenStax, free): chapter 3 (functions; §3.5 moves and stretches graphs), chapter 4 (linear functions) and chapter 6 (exponential and logarithmic functions), with many exercises and answers.
• Deisenroth, Faisal and Ong, Mathematics for Machine Learning (free PDF): §8.3 uses the negative log-likelihood for parameter estimation.
• Kaplan et al. (2020), Scaling Laws for Neural Language Models: figure 1 shows power laws as straight lines on log-log plots.