Q:

Functional equation $ f(x+yf(x^2))=f(x)+xf(xy) $

I have trouble solving the following functional equation: $f:\mathbb{R}\to\mathbb{R}$ such that $f(x+yf(x^2))=f(x)+xf(xy)$.
I think $f(x)=x$ is the unique solution, especially, $f(x+y)=f(x)+f(y)$. But how do we show that?

A:

You can show that the only continuous solutions to the functional equation
$$ f \Big( x + y f \big( x ^ 2 \big) \Big) = f ( x ) + x f ( x y ) \tag 0 \label 0 $$
are the constant zero function and the identity function. It's easy to see that these two are solutions. To show that they are the only ones, first let $ y = 0 $ in \eqref{0} to get $ f ( 0 ) = 0 $. Now, assuming $ f ( 1 ) \ne 1 $, you can let $ x = 1 $ and $ y = \frac 1 { 1 - f ( 1 ) } $ in \eqref{0} which leads to $ f ( 1 ) = 0 $. Thus, we must either have $ f ( 1 ) = 0 $ or $ f ( 1 ) = 1 $. In the case that $ f ( 1 ) = 0 $, you can let $ x = 1 $ in \eqref{0} and conclude that $ f $ is the constant zero function. From now on, we assume that $ f ( 1 ) = 1 $. Letting $ x = 1 $ in \eqref{0}, you get $ f ( y + 1 ) = f ( y ) + 1 $. Substituting $ y - 1 $ for $ y $ in this equation, we get $ f ( y - 1 ) = f ( y ) - 1 $. Using the last two equations, we inductively get
$$ f ( x + n ) = f ( x ) + n \tag 1 \label 1 $$
for each integer $ n $. In particular, this shows that $ f ( n ) = n $ for any integer $ n $. Thus, if for some integer $ n $ we let $ x = n $ in \eqref{0}, we'll have
$ f \big( n ^ 2 y \big) = n f ( n y ) $. Letting $ y = \frac m { n ^ 2 } $, where $ m $ and $ n $ are integers with $ n \ne 0 $, we get
$$ f \Big( \frac m n \Big) = \frac m n \text . \tag 2 \label 2 $$
Since the set of rational numbers is dense in the set of real numbers, if $ f $ is continuous, then by \eqref{2} we can show that $ f $ is the identity function. 

A:

The beginning steps are the same as in the answer of @MohsenShahriari, which I repeat for sake of completeness.

Our equation is $$f(x + yf(x^2)) = f(x) + xf(xy).\tag 0$$
Setting $y = 0$ in $(0)$, we get $xf(0) = 0$. Setting again $x = 1$ gives us $f(0) = 0$.
Setting $x = 1$ in $(0)$, we get $$f(1 + yf(1)) = f(1) + f(y).\tag 1$$
If $f(1) \neq 1$, then we may set $y = \frac{1}{1 - f(1)}$ in $(1)$ and get $f(1) = 0$. Then $(1)$ implies that $f(y) = 0$ for all $y\in \Bbb R$, which is one solution to the equation.
Now assume that $f(1) = 1$. Then $(1)$ becomes $$f(1 + y) = 1 + f(y).\tag 2$$ 
Using induction with $(2)$ gives $$f(n + y) = n + f(y), \forall n\in \Bbb Z_{> 0}.\tag 3$$
Setting $y = y - n$ in $(3)$, we get $f(-n + y) = -n + f(y), \forall n \in \Bbb Z_{> 0}$. Combined with $(3)$, we have $$f(n + y) = n + f(y), \forall n\in \Bbb Z.\tag 4$$
Setting $y = 0$ in $(4)$, we get $f(n) = n, \forall n \in \Bbb Z$.
For integer $n \neq 0$, setting $x = n$ and $y = \frac y n$ in $(0)$ gives $n + f(ny) = n + nf(y)$, hence we get $f(ny) = nf(y)$. In particular, for every rational number $q = \frac u v$ with $u, v \in \Bbb Z$, setting $y = q$ and $n = v$ gives $f(q) = q$.
Setting $x = -x$ in $(0)$, we get $$f(-x + y f(x^2)) = f(-x) - xf(-xy) = -f(x) + xf(xy).\tag 5$$
Taking difference of $(0)$ and $(5)$, we get $$f(x + yf(x^2)) - f(-x + yf(x^2)) = 2f(x).\tag 6$$
Now for any $t\neq 0$, we have $f(t^2)\neq 0$, otherwise setting $x = t$ and $y = \frac 1 t$ in $(0)$ would give $f(1) = 0$, contradicting our hypothesis.
Therefore, if $x\neq 0$, then we may set $y = \frac{x + y}{f(x^2)}$ in $(6)$ and get $f(2x + y) - f(y) = 2f(x) = f(2x)$. Setting again $x = \frac x 2$, we get $$f(x + y) = f(x) + f(y).\tag 7$$
Using $(7)$, we may rewrite $(0)$ and get $$f(yf(x^2)) = xf(xy).\tag 8$$
Setting $y = x$ in $(8)$, we get $f(xf(x^2)) = xf(x^2)$.
Setting $y = f(x^2)$ in $(8)$, we get $f(f(x^2)^2) = xf(xf(x^2)) = x^2f(x^2)$. Since $x^2$ can be any non-negative real number, we have $$f(f(x)^2) = xf(x), \forall x \geq 0. \tag 9$$
Setting $x = -x$ in $(9)$, we get $f(f(x)^2) = xf(x), \forall x \leq 0$. Combined with $(9)$, we get $f(f(x)^2) = xf(x), \forall x$.
Setting $x = f(x)$ in $(8)$, we get $f(yf(f(x)^2)) = f(x)f(f(x)y)$, hence $f(xyf(x)) = f(x)f(yf(x))$. When $f(x) \neq 0$, setting again $y = \frac y {f(x)}$ gives $f(xy) = f(x)f(y)$.
When $f(x) = 0$, we note that $f(x + 1) = f(x) + f(1) = 1 \neq 0$, hence we have $f((x + 1)y) = f(x + 1) f(y)$, which again leads to $f(xy) = f(x)f(y)$. Therefore we have $f(xy) = f(x)f(y)$ for all $x, y$.
It follows that $f$ is an increasing function. In fact, for any $x \geq 0$, we have $f(x) = f(\sqrt x)^2 \geq 0$. Hence whenever $x \geq y$, we have $f(x) - f(y) = f(x - y) \geq 0$.
Since we already know $f(q) = q$ for any rational number $q$, this implies $f(x) = x$ for any real number $x$.
Otherwise, assume there is $x$ such that $f(x) \neq x$. Replacing $x$ with $-x$ if necessary, we may assume that $f(x) > x$. Then there is a rational number $q$ such that $f(x) > q > x$. But since $f$ is increasing, $q > x$ implies $q = f(q) > f(x)$, contradiction.

