解決

column_names = User.column_names

models = []
# datasには[{id: 1, name: 'hoge'...}...]
datas.each do |data|
  User.find_or_initialize_by(id: data[:id]) do |model|
    # FIXME: In this case, new_record is not working
    model.assign_attributes(data)
    models << model
  end
end

User.import(models, on_duplicate_key_update: { conflict_target: %i(id), columns: column_names })
new_record?メソッドで新規作成か判定しようとしたが、全て新規作成となる(バグ?)

zdennis/activerecord-import: A library for bulk insertion of data into your database using ActiveRecord. [Rails]find_or_create_byとfind_or_initialize_by Railsで一括でデータを更新する · aguuu.com