The second (desired) example uses a different font family. They define thin brackets. I am not a font expert, so I can’t say which font uses this example. I can tell you, I know that the Math-Fourier font also uses thin brackets. This is what I used to recreate your second example:
\documentclass[english]{article} \usepackage{fourier} % The Math font \usepackage[T1]{fontenc} \usepackage[latin9]{inputenc} \usepackage{babel} \begin{document} \[ F(x)={\scriptstyle \left(\begin{array}{cc} {\textstyle \frac{x}{2}} & {\displaystyle 0}\\ \vphantom{{\displaystyle W}} % uses some vertical spacing % between the first and second row {\displaystyle 0} & {\displaystyle x^{2}}\end{array}\right)}\] \end{document}