Conventional reverse osmosis (RO) systems are widely used for the effective desalination of brackish water and seawater for various purposes; however, they require substantial energy for desalination. Therefore, we adopted a sodium metal electrode-based hybrid redox flow battery desalination (NRFDB-D) system. This system is a modified seawater battery desalination (SWB-D) systems which utilizes a redox catholyte to reduce energy consumption. The energy consumption of the NRFDB-D system can be decreased by increasing the catholyte concentration; however, this can simultaneously lead to osmosis-induced product loss. Therefore, the NRFDB-D system was investigated in terms of catholyte concentration and osmosis through experiments and modeling, resulting in minimized specific energy consumption (SEC). The relationship between the catholyte concentration and osmosis in the NRFDB-D system was determined in the NRFDB-D experiment. Response surface methodology (RSM) analysis provided the optimal operating conditions, yielding the lowest SEC across various feedwater concentrations and salt removal rates. The optimized NRFDB-D system efficiently desalinated low-salt water (12,500 mg/L) and seawater (35,064 mg/L), exhibiting a lower SEC than that of the RO system. Furthermore, using the optimized NRFDB-D as pretreatment (45 % salt removal) for the RO system offset the limitations of the individual systems, including concentrated brine generation and divalent ion removal from the feedwater.