Should I keep the calories up on non-fast days in order for metabolism to not slow down?

Unfortunately, all weight loss diets lower your metabolism by 10-15% when you reduce your body weight by 10% or more. This occurs with both daily calorie restriction and intermittent fasting. The only way to prevent this drop in metabolic rate is to exercise 5 times per week for 45 min/session (which increases your muscle mass). Since your muscle mass is a key determinant of your metabolic rate, increasing muscle mass will ensure your metabolism stays high, even post-weight loss.