Fractal geometry is a field of math born in the 1970s and mainly developed by Benoit Mandelbrot. If you've already heard of fractals, you've probably seen the picture above. It's called the Mandelbrot Set and is an example of a fractal shape.
The geometry that you learned in school was about how to make shapes; fractal geometry is no different. While the shapes that you learned in classical geometry were 'smooth', such as a circle or a triangle, the shapes that come out of fractal geometry are 'rough' and infinitely complex.