Our approach for fitting dynamic nonstationary factor models to multivariate time series is based on the principal components of the estimated time-varying spectral-density matrix. This approach allows the spectral matrix to be smoothly time-varying, which imposes very little structure on the moments of the underlying process. However, the estimation delivers time-varying filters that are high-dimensional and two-sided. Moreover, the estimation of the spectral matrix strongly depends on the chosen bandwidths for smoothing over frequency and time. As an alternative, we propose a new semi-parametric approach in which only part of the model is allowed to be time-varying. More precisely, the latent factors admit a dynamic representation with time-varying autoregressive coefficients while the loadings are constant over time. Estimation of the model parameters is accomplished by application of the EM algorithm and the Kalman filter. The time-varying parameters are modeled locally by polynomials and estimated by maximizing the likelihood locally. Compared to estimation of the factors by principal components, our new approach produces superior results in particular for small cross-sectional dimension.