Factorial(N) gets very, very big even for small N. It soon gets too big to hold in a Java int and wrong answers result. In fact, 12 is the largest int that will work as an argument.
You could improve the Java method by using long or perhaps double in place of int. This would extend the range of arguments that could be used with it, but the range is not extended very far.
Usually in a practical situatuion, when you see factorial in a formula you should not actually implement it with a Java method. The correct approach is to manipulate the formula until factorial is removed. Usually this makes the formula less pretty, but makes it suitable for implementation in a program. Topics like these are the subject of numerical analysis, a common course in a Computer Science curriculum.