It's possible to set custom resolution screen which is not my monitor native? Ex: My monitor is 1366x768 and I set 1920x1080.
What Terrance said. Your monitor likely will not do it and display either a blank screen or a jumbled mess. You might or might not be able to recover, depending upon whether the monitor survives the abuse.
Easiest way to play with your monitor's capabilities is under Windows -- anything from XP on up. Right click on the desktop and follow the links to monitor settings. After you change settings, you will get a yes/no dialog box to keep that resolution. If your experiment fails and you can't see the box, Windows will revert to the previous setting after a few seconds.
If Windows recognizes the make and model of your monitor, you might not be allowed to set a resolution outsied the monitor's known capabilities.
Again, as Terrance said, you could fry the monitor hardware, depending on how good its built-in abuse defenses are.
You can do it, but the monitor is likely to interpolate/extrapolate the given resolution to the monitor's native resolution. Its bounds depend on the capabilities of the monitor.
At least, this is how many LCD/LED monitors do behave. Old CRT monitors may be damaged by setting a not supported resolution.